2024-03-18T15:38:30.674 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T15:38:30.675 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_pdonnell@teuthology --verbose --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905 -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/orig.config.yaml 2024-03-18T15:38:30.728 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T15:38:30.818 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608905' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWnDLCEvrTGwexuUVs+s2aEBzskPO31yExEk/1HVtP2AMtyfWTvwDCuQ20GIOpbUaCcZTpsorSh3uQhG7Hk6Pw= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqo8HF0mk5hVSYWkfZ0kSgaCgVo0YQEc2mLFsYAhEzuKq1GtgSJBUyWPbCBfhCKPF5asZwMUoY3PuJfNSJUvlg= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvpS9p7oSdf1WzK50BNQGRK/xrBY1V9Pjr5s+PsteXPEGx21njDYhacZi6D6ksPvMGCOG2PIn9jTZnmzTvg/Pc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T15:38:30.818 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T15:38:30.819 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T15:38:30.819 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T15:38:30.820 INFO:teuthology.task.internal:Checking packages... 2024-03-18T15:38:30.842 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T15:38:30.842 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T15:38:30.842 INFO:teuthology.packaging:ref: None 2024-03-18T15:38:30.842 INFO:teuthology.packaging:tag: None 2024-03-18T15:38:30.842 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T15:38:30.842 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:38:30.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T15:38:30.980 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T15:38:30.981 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T15:38:30.989 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T15:38:30.989 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T15:38:31.031 INFO:teuthology.task.internal:Saving configuration 2024-03-18T15:38:31.050 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T15:38:31.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:38:31.082 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.191917', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWnDLCEvrTGwexuUVs+s2aEBzskPO31yExEk/1HVtP2AMtyfWTvwDCuQ20GIOpbUaCcZTpsorSh3uQhG7Hk6Pw='} 2024-03-18T15:38:31.107 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.190322', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqo8HF0mk5hVSYWkfZ0kSgaCgVo0YQEc2mLFsYAhEzuKq1GtgSJBUyWPbCBfhCKPF5asZwMUoY3PuJfNSJUvlg='} 2024-03-18T15:38:31.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.188733', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvpS9p7oSdf1WzK50BNQGRK/xrBY1V9Pjr5s+PsteXPEGx21njDYhacZi6D6ksPvMGCOG2PIn9jTZnmzTvg/Pc='} 2024-03-18T15:38:31.130 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T15:38:31.137 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T15:38:31.137 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T15:38:31.137 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T15:38:31.137 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T15:38:31.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc3f4c02160>, signals=[15]) 2024-03-18T15:38:31.229 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T15:38:31.236 INFO:teuthology.task.internal:Opening connections... 2024-03-18T15:38:31.236 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T15:38:31.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:38:31.329 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-03-18T15:38:31.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:38:31.418 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-18T15:38:31.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:38:31.507 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T15:38:31.514 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-03-18T15:38:31.532 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-03-18T15:38:31.533 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-03-18T15:38:31.589 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-03-18T15:38:31.589 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-03-18T15:38:31.589 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-03-18T15:38:31.589 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:38:31.590 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:38:31.591 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-03-18T15:38:31.614 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-03-18T15:38:31.632 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-03-18T15:38:31.632 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-03-18T15:38:31.688 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:38:31.689 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:38:31.690 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-03-18T15:38:31.713 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-18T15:38:31.731 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-18T15:38:31.731 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-18T15:38:31.788 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-18T15:38:31.788 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-18T15:38:31.788 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-18T15:38:31.788 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:38:31.788 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:38:31.789 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:38:31.789 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-18T15:38:31.804 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T15:38:31.815 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T15:38:31.821 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T15:38:31.821 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:38:31.824 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:38:31.826 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:38:31.844 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T15:38:31.851 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T15:38:31.851 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:38:31.881 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:38:31.883 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:38:31.979 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T15:38:32.058 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T15:38:32.072 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T15:38:32.072 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-18T15:38:32.072 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-18T15:38:32.072 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:38:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:38:32.089 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-18T15:38:32.146 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-03-18T15:38:32.146 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-03-18T15:38:32.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-03-18T15:38:34.297 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T15:38:35.905 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-18T15:38:36.754 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-18T15:38:38.841 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-18T15:38:43.248 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.1 MB 00:00 2024-03-18T15:38:45.690 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-18T15:38:47.341 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-18T15:38:52.400 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 7.4 kB/s | 1.7 kB 00:00 2024-03-18T15:38:54.623 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T15:38:54.623 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T15:38:54.703 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T15:38:54.710 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:38:54.711 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 100 M 2024-03-18T15:38:54.712 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 123 M 2024-03-18T15:38:54.712 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-18T15:38:56.284 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-18T15:39:07.217 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 3.1 MB/s | 33 MB 00:10 2024-03-18T15:39:14.300 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.1 MB/s | 21 MB 00:19 2024-03-18T15:39:23.574 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.4 MB/s | 39 MB 00:28 2024-03-18T15:39:23.574 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:39:23.575 INFO:teuthology.orchestra.run.smithi150.stdout:Total 3.4 MB/s | 100 MB 00:28 2024-03-18T15:39:24.488 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-18T15:39:24.874 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-18T15:39:24.874 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-18T15:39:26.198 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-18T15:39:26.198 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-18T15:39:29.160 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-18T15:39:29.829 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T15:39:29.958 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T15:39:32.769 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T15:39:32.943 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T15:39:37.691 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T15:39:37.886 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:39:42.640 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:40:11.972 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:40:11.973 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T15:40:11.973 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:40:12.058 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:40:12.517 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:40:12.517 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T15:40:12.517 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T15:40:12.518 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T15:40:12.929 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:40:12.930 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-18T15:40:13.443 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T15:40:13.586 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T15:40:13.587 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T15:40:13.587 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T15:40:13.587 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T15:40:13.587 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T15:40:13.587 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-18T15:40:13.587 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T15:40:13.587 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T15:40:13.708 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T15:40:13.709 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T15:40:13.709 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-03-18T15:40:13.743 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:40:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T15:40:13.745 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T15:40:13.745 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-427.el9.x86_64 2024-03-18T15:40:13.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T15:40:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-03-18T15:40:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T15:40:15.857 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-03-18T15:40:15.860 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-03-18T15:40:15.876 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.pV3Th2uycv 2024-03-18T15:40:15.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pV3Th2uycv 2024-03-18T15:40:15.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.pV3Th2uycv 2024-03-18T15:40:16.085 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.pV3Th2uycv is 6KB 2024-03-18T15:40:16.105 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.pV3Th2uycv 2024-03-18T15:40:16.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T15:40:16.183 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T15:40:16.183 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T15:40:16.183 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T15:40:16.183 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T15:40:16.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T15:40:16.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-03-18T15:40:46.347 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T15:40:46.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:40:46.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:41:04.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-18T15:41:13.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:41:13.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:41:16.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-18T15:41:28.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:41:28.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:41:47.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-18T15:42:02.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:42:02.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:42:02.514 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-18T15:42:03.163 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:42:03.163 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-18T15:42:03.163 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-18T15:42:03.177 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-427.el9.x86_64 2024-03-18T15:42:03.178 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T15:42:03.178 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T15:42:03.178 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T15:42:04.179 DEBUG:teuthology.parallel:result is None 2024-03-18T15:42:04.180 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T15:42:04.187 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T15:42:04.188 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:42:04.190 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:42:04.192 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:42:04.211 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T15:42:04.217 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T15:42:04.226 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T15:42:04.226 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:42:04.245 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:42:04.250 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:42:04.389 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T15:42:04.396 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T15:42:04.396 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:42:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:04.413 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:42:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:04.430 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:42:04.432 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:42:04.456 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:42:04.515 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.516 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.533 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.539 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.540 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.579 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:42:04.581 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T15:42:04.588 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T15:42:04.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:42:04.591 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:42:04.593 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:42:04.623 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T15:42:04.631 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T15:42:04.631 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:42:04.658 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:42:04.663 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:42:04.679 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:04.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:04.802 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:04.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:04.927 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:42:04.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:42:05.015 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:05.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:05.119 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:05.163 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:05.206 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:42:05.206 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:42:05.275 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:05.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:42:05.376 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:05.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:42:05.506 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:42:05.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:42:05.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-03-18T15:42:05.577 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-03-18T15:42:05.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-18T15:42:05.615 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:42:05.618 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:42:05.655 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:42:06.048 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T15:42:06.056 INFO:teuthology.task.internal:Starting timer... 2024-03-18T15:42:06.056 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T15:42:06.065 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T15:42:06.074 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T15:42:06.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-03-18T15:42:06.129 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-03-18T15:42:06.130 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-03-18T15:42:06.182 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-03-18T15:42:06.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-18T15:42:06.262 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-18T15:42:06.264 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T15:42:06.264 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-03-18T15:42:06.286 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-03-18T15:42:06.286 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-03-18T15:42:06.318 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-03-18T15:42:06.318 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-18T15:42:06.338 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-18T15:42:06.338 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-18T15:42:06.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:42:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:06.367 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:42:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:06.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:42:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:06.424 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T15:42:06.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:42:06.488 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:42:06.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:42:06.555 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T15:42:06.564 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T15:42:06.570 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T15:42:06.694 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T15:42:06.709 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T15:42:06.710 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com,smithi156.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T15:47:02.295 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-18T15:47:02.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:47:02.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:47:02.386 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-18T15:47:02.465 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-18T15:47:02.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-18T15:47:02.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:47:02.555 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-18T15:47:02.621 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-18T15:47:02.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-18T15:47:02.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:47:02.716 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-18T15:47:02.791 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-18T15:47:02.791 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T15:47:02.801 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T15:47:02.801 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:47:02.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:47:02.804 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:47:02.804 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:47:02.807 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:47:02.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:47:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:47:02.836 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:47:02.850 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:47:02.853 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:47:02.866 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:47:02.879 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-03-18T15:47:02.882 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-03-18T15:47:02.884 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:47:02.893 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-03-18T15:47:02.895 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-03-18T15:47:02.910 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:47:02.912 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:47:02.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-18T15:47:02.926 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:47:02.928 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-18T15:47:02.928 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:47:02.945 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:47:02.962 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:47:02.972 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:47:02.975 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:47:02.975 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-18T15:47:02.975 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.975 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.976 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.976 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.978 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:02.981 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:03.006 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:47:03.009 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:47:03.010 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-18T15:47:03.010 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:03.010 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:03.010 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:03.010 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:47:03.011 INFO:teuthology.run_tasks:Running task install... 2024-03-18T15:47:03.020 DEBUG:teuthology.task.install:project ceph 2024-03-18T15:47:03.020 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T15:47:03.020 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T15:47:03.020 INFO:teuthology.task.install:Using flavor: default 2024-03-18T15:47:03.028 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T15:47:03.028 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T15:47:03.028 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:47:03.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:47:03.031 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:47:03.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:47:03.032 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:47:03.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:47:03.176 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:47:03.177 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:47:03.184 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:47:03.184 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:47:03.291 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:47:03.292 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:47:03.292 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:47:03.293 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:47:03.293 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:47:03.298 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:47:03.299 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:47:03.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:47:03.326 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:47:03.327 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:47:03.329 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:47:03.329 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:47:03.395 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:47:03.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:47:03.407 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:47:03.407 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:47:03.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:47:03.437 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:47:03.438 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:47:03.475 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:47:03.477 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:47:03.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:47:03.542 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-03-18T15:47:03.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-03-18T15:47:03.544 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-18T15:47:03.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-18T15:47:03.589 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:47:03.655 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-03-18T15:47:03.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-03-18T15:47:03.778 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-18T15:47:03.779 INFO:teuthology.orchestra.run.smithi156.stdout:74 files removed 2024-03-18T15:47:03.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:47:03.807 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:47:03.898 INFO:teuthology.orchestra.run.smithi150.stdout:74 files removed 2024-03-18T15:47:03.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:47:04.519 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 195 kB/s | 83 kB 00:00 2024-03-18T15:47:04.555 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-03-18T15:47:04.655 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-03-18T15:47:04.955 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-18T15:47:04.987 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-18T15:47:05.064 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-18T15:47:05.264 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-18T15:47:05.405 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-18T15:47:05.456 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-18T15:47:05.766 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-18T15:47:06.108 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-18T15:47:06.192 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-18T15:47:07.259 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-18T15:47:07.624 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-18T15:47:07.651 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-18T15:47:08.291 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-18T15:47:09.209 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-18T15:47:09.258 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-18T15:47:11.256 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-18T15:47:13.231 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-03-18T15:47:14.503 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 19 MB 00:05 2024-03-18T15:47:15.194 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-18T15:47:16.654 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-18T15:47:18.264 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-18T15:47:18.370 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-18T15:47:19.954 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-18T15:47:20.920 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-18T15:47:21.931 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T15:47:22.301 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 16 kB 00:00 2024-03-18T15:47:23.239 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-18T15:47:23.613 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-18T15:47:25.736 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:25.737 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:25.751 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:47:25.752 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:47:25.754 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:47:25.754 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:25.756 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:25.760 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:47:25.760 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:47:25.840 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-18T15:47:25.864 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-18T15:47:25.864 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-18T15:47:25.864 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-18T15:47:25.864 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:47:25.865 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:47:25.866 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:47:25.867 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:47:25.868 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:47:25.869 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:47:25.870 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:47:25.871 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:47:25.872 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:47:25.873 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:47:25.874 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:47:25.875 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:47:25.876 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout:Install 150 Packages 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 24 Packages 2024-03-18T15:47:25.877 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:47:25.878 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 224 M 2024-03-18T15:47:25.878 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-18T15:47:26.616 INFO:teuthology.orchestra.run.smithi191.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-18T15:47:26.792 INFO:teuthology.orchestra.run.smithi191.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 5.0 MB/s | 890 kB 00:00 2024-03-18T15:47:26.842 INFO:teuthology.orchestra.run.smithi191.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.8 MB/s | 145 kB 00:00 2024-03-18T15:47:26.847 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-18T15:47:27.042 INFO:teuthology.orchestra.run.smithi191.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 12 MB/s | 2.4 MB 00:00 2024-03-18T15:47:27.159 INFO:teuthology.orchestra.run.smithi191.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-18T15:47:27.418 INFO:teuthology.orchestra.run.smithi191.stdout:(6/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 18 MB/s | 4.7 MB 00:00 2024-03-18T15:47:28.260 INFO:teuthology.orchestra.run.smithi191.stdout:(7/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 20 MB/s | 17 MB 00:00 2024-03-18T15:47:28.635 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-18T15:47:28.794 INFO:teuthology.orchestra.run.smithi191.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 18 MB/s | 9.8 MB 00:00 2024-03-18T15:47:28.836 INFO:teuthology.orchestra.run.smithi191.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 603 kB/s | 25 kB 00:00 2024-03-18T15:47:29.286 INFO:teuthology.orchestra.run.smithi191.stdout:(10/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x 1.9 MB/s | 5.4 MB 00:02 2024-03-18T15:47:29.336 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:29.337 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:29.351 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:47:29.352 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:47:29.353 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:47:29.354 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:29.356 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:29.359 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:47:29.360 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:47:29.440 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:47:29.463 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:47:29.464 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:47:29.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:47:29.466 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:47:29.467 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:47:29.468 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:47:29.469 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:47:29.470 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:47:29.471 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:47:29.474 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:47:29.475 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:47:29.476 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:47:29.477 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:47:29.478 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout:Install 150 Packages 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 24 Packages 2024-03-18T15:47:29.479 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:47:29.480 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 224 M 2024-03-18T15:47:29.480 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-18T15:47:29.503 INFO:teuthology.orchestra.run.smithi191.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 221 kB 00:00 2024-03-18T15:47:29.612 INFO:teuthology.orchestra.run.smithi191.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 249 kB/s | 27 kB 00:00 2024-03-18T15:47:29.704 INFO:teuthology.orchestra.run.smithi191.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 346 kB/s | 32 kB 00:00 2024-03-18T15:47:30.038 INFO:teuthology.orchestra.run.smithi191.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.2 MB/s | 747 kB 00:00 2024-03-18T15:47:30.146 INFO:teuthology.orchestra.run.smithi191.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.5 MB/s | 163 kB 00:00 2024-03-18T15:47:30.247 INFO:teuthology.orchestra.run.smithi191.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.2 MB/s | 127 kB 00:00 2024-03-18T15:47:30.514 INFO:teuthology.orchestra.run.smithi191.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 1.8 MB/s | 501 kB 00:00 2024-03-18T15:47:30.932 INFO:teuthology.orchestra.run.smithi150.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-18T15:47:31.116 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:31.117 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:47:31.131 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:47:31.132 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:47:31.133 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:47:31.134 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:31.136 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:47:31.139 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:47:31.140 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:47:31.220 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:47:31.239 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:47:31.240 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:47:31.241 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:47:31.244 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:47:31.245 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:47:31.246 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:47:31.247 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:47:31.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:47:31.249 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:47:31.250 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:47:31.251 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout:Install 150 Packages 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 24 Packages 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 224 M 2024-03-18T15:47:31.252 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-18T15:47:31.356 INFO:teuthology.orchestra.run.smithi191.stdout:(18/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 19 MB/s | 48 MB 00:02 2024-03-18T15:47:31.431 INFO:teuthology.orchestra.run.smithi191.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 604 kB/s | 45 kB 00:00 2024-03-18T15:47:31.523 INFO:teuthology.orchestra.run.smithi191.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.4 MB/s | 131 kB 00:00 2024-03-18T15:47:31.617 INFO:teuthology.orchestra.run.smithi191.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 162 kB 00:00 2024-03-18T15:47:31.691 INFO:teuthology.orchestra.run.smithi191.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.2 MB/s | 320 kB 00:00 2024-03-18T15:47:31.776 INFO:teuthology.orchestra.run.smithi191.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-18T15:47:31.876 INFO:teuthology.orchestra.run.smithi191.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 995 kB/s | 99 kB 00:00 2024-03-18T15:47:31.957 INFO:teuthology.orchestra.run.smithi150.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 868 kB/s | 890 kB 00:01 2024-03-18T15:47:31.968 INFO:teuthology.orchestra.run.smithi191.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 930 kB/s | 85 kB 00:00 2024-03-18T15:47:32.182 INFO:teuthology.orchestra.run.smithi150.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 644 kB/s | 145 kB 00:00 2024-03-18T15:47:32.262 INFO:teuthology.orchestra.run.smithi156.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-18T15:47:32.535 INFO:teuthology.orchestra.run.smithi191.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 5.5 MB/s | 3.1 MB 00:00 2024-03-18T15:47:32.602 INFO:teuthology.orchestra.run.smithi191.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-18T15:47:32.660 INFO:teuthology.orchestra.run.smithi191.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 583 kB/s | 34 kB 00:00 2024-03-18T15:47:32.752 INFO:teuthology.orchestra.run.smithi191.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-18T15:47:32.861 INFO:teuthology.orchestra.run.smithi191.stdout:(30/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 2.2 MB/s | 5.1 MB 00:02 2024-03-18T15:47:33.228 INFO:teuthology.orchestra.run.smithi191.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 3.8 MB/s | 1.8 MB 00:00 2024-03-18T15:47:33.338 INFO:teuthology.orchestra.run.smithi191.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.2 MB/s | 248 kB 00:00 2024-03-18T15:47:33.422 INFO:teuthology.orchestra.run.smithi191.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 591 kB/s | 49 kB 00:00 2024-03-18T15:47:33.537 INFO:teuthology.orchestra.run.smithi191.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 144 kB/s | 16 kB 00:00 2024-03-18T15:47:33.639 INFO:teuthology.orchestra.run.smithi191.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.5 MB/s | 264 kB 00:00 2024-03-18T15:47:33.678 INFO:teuthology.orchestra.run.smithi156.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 628 kB/s | 890 kB 00:01 2024-03-18T15:47:33.766 INFO:teuthology.orchestra.run.smithi150.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-03-18T15:47:33.898 INFO:teuthology.orchestra.run.smithi191.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.8 MB/s | 748 kB 00:00 2024-03-18T15:47:33.990 INFO:teuthology.orchestra.run.smithi191.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 199 kB/s | 18 kB 00:00 2024-03-18T15:47:34.307 INFO:teuthology.orchestra.run.smithi191.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2024-03-18T15:47:34.441 INFO:teuthology.orchestra.run.smithi191.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-18T15:47:34.479 INFO:teuthology.orchestra.run.smithi156.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 181 kB/s | 145 kB 00:00 2024-03-18T15:47:34.499 INFO:teuthology.orchestra.run.smithi191.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-18T15:47:34.558 INFO:teuthology.orchestra.run.smithi191.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-18T15:47:34.591 INFO:teuthology.orchestra.run.smithi191.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 953 kB/s | 32 kB 00:00 2024-03-18T15:47:34.758 INFO:teuthology.orchestra.run.smithi191.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-18T15:47:34.809 INFO:teuthology.orchestra.run.smithi191.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-18T15:47:34.816 INFO:teuthology.orchestra.run.smithi150.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-03-18T15:47:34.859 INFO:teuthology.orchestra.run.smithi191.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 202 kB/s | 10 kB 00:00 2024-03-18T15:47:34.909 INFO:teuthology.orchestra.run.smithi191.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T15:47:34.960 INFO:teuthology.orchestra.run.smithi191.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-18T15:47:35.027 INFO:teuthology.orchestra.run.smithi191.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-18T15:47:35.094 INFO:teuthology.orchestra.run.smithi191.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-18T15:47:35.144 INFO:teuthology.orchestra.run.smithi191.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 926 kB/s | 46 kB 00:00 2024-03-18T15:47:35.211 INFO:teuthology.orchestra.run.smithi191.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-18T15:47:35.269 INFO:teuthology.orchestra.run.smithi191.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-18T15:47:35.336 INFO:teuthology.orchestra.run.smithi191.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-18T15:47:35.350 INFO:teuthology.orchestra.run.smithi150.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 5.4 MB 00:04 2024-03-18T15:47:35.370 INFO:teuthology.orchestra.run.smithi156.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-03-18T15:47:35.386 INFO:teuthology.orchestra.run.smithi191.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-18T15:47:35.617 INFO:teuthology.orchestra.run.smithi150.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 21 MB 00:05 2024-03-18T15:47:35.645 INFO:teuthology.orchestra.run.smithi191.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-18T15:47:35.746 INFO:teuthology.orchestra.run.smithi191.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-18T15:47:35.829 INFO:teuthology.orchestra.run.smithi191.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 9.5 MB/s | 811 kB 00:00 2024-03-18T15:47:35.954 INFO:teuthology.orchestra.run.smithi156.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-18T15:47:36.005 INFO:teuthology.orchestra.run.smithi191.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 764 kB/s | 134 kB 00:00 2024-03-18T15:47:36.372 INFO:teuthology.orchestra.run.smithi191.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 15 MB/s | 5.7 MB 00:00 2024-03-18T15:47:36.497 INFO:teuthology.orchestra.run.smithi191.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-03-18T15:47:36.564 INFO:teuthology.orchestra.run.smithi191.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 78 kB 00:00 2024-03-18T15:47:36.864 INFO:teuthology.orchestra.run.smithi191.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-18T15:47:36.893 INFO:teuthology.orchestra.run.smithi191.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 986 kB/s | 24 kB 00:00 2024-03-18T15:47:36.915 INFO:teuthology.orchestra.run.smithi191.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 723 kB/s | 18 kB 00:00 2024-03-18T15:47:36.965 INFO:teuthology.orchestra.run.smithi191.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T15:47:36.999 INFO:teuthology.orchestra.run.smithi191.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 948 kB/s | 32 kB 00:00 2024-03-18T15:47:37.187 INFO:teuthology.orchestra.run.smithi156.stdout:(6/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 3.9 MB/s | 21 MB 00:05 2024-03-18T15:47:37.216 INFO:teuthology.orchestra.run.smithi191.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-18T15:47:37.274 INFO:teuthology.orchestra.run.smithi191.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 217 kB 00:00 2024-03-18T15:47:37.325 INFO:teuthology.orchestra.run.smithi191.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T15:47:37.367 INFO:teuthology.orchestra.run.smithi191.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T15:47:37.400 INFO:teuthology.orchestra.run.smithi191.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T15:47:37.450 INFO:teuthology.orchestra.run.smithi191.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T15:47:37.501 INFO:teuthology.orchestra.run.smithi191.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T15:47:37.560 INFO:teuthology.orchestra.run.smithi191.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-18T15:47:37.601 INFO:teuthology.orchestra.run.smithi191.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T15:47:37.652 INFO:teuthology.orchestra.run.smithi191.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-18T15:47:37.867 INFO:teuthology.orchestra.run.smithi150.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 4.4 MB/s | 9.8 MB 00:02 2024-03-18T15:47:37.967 INFO:teuthology.orchestra.run.smithi150.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 252 kB/s | 25 kB 00:00 2024-03-18T15:47:38.013 INFO:teuthology.orchestra.run.smithi156.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-18T15:47:38.179 INFO:teuthology.orchestra.run.smithi156.stdout:(8/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 653 kB/s | 2.4 MB 00:03 2024-03-18T15:47:38.252 INFO:teuthology.orchestra.run.smithi191.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-18T15:47:38.285 INFO:teuthology.orchestra.run.smithi191.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-18T15:47:38.288 INFO:teuthology.orchestra.run.smithi156.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 233 kB/s | 25 kB 00:00 2024-03-18T15:47:38.328 INFO:teuthology.orchestra.run.smithi191.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 964 kB/s | 41 kB 00:00 2024-03-18T15:47:38.370 INFO:teuthology.orchestra.run.smithi191.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-18T15:47:38.403 INFO:teuthology.orchestra.run.smithi191.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-03-18T15:47:38.438 INFO:teuthology.orchestra.run.smithi191.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T15:47:38.471 INFO:teuthology.orchestra.run.smithi191.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T15:47:38.504 INFO:teuthology.orchestra.run.smithi191.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-18T15:47:38.546 INFO:teuthology.orchestra.run.smithi191.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-18T15:47:38.639 INFO:teuthology.orchestra.run.smithi191.stdout:(86/174): ceph-mgr-diskprediction-local-19.0.0- 1.3 MB/s | 7.4 MB 00:05 2024-03-18T15:47:38.680 INFO:teuthology.orchestra.run.smithi191.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.1 MB 00:00 2024-03-18T15:47:38.706 INFO:teuthology.orchestra.run.smithi191.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-18T15:47:38.748 INFO:teuthology.orchestra.run.smithi191.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 124 kB 00:00 2024-03-18T15:47:38.790 INFO:teuthology.orchestra.run.smithi191.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-18T15:47:38.809 INFO:teuthology.orchestra.run.smithi150.stdout:(10/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 1.2 MB/s | 4.7 MB 00:03 2024-03-18T15:47:39.040 INFO:teuthology.orchestra.run.smithi191.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 642 kB/s | 187 kB 00:00 2024-03-18T15:47:39.134 INFO:teuthology.orchestra.run.smithi150.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 682 kB/s | 221 kB 00:00 2024-03-18T15:47:39.157 INFO:teuthology.orchestra.run.smithi191.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-18T15:47:39.182 INFO:teuthology.orchestra.run.smithi191.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-18T15:47:39.266 INFO:teuthology.orchestra.run.smithi191.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-18T15:47:39.267 INFO:teuthology.orchestra.run.smithi150.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 203 kB/s | 27 kB 00:00 2024-03-18T15:47:39.283 INFO:teuthology.orchestra.run.smithi191.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-03-18T15:47:39.300 INFO:teuthology.orchestra.run.smithi191.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 758 kB/s | 25 kB 00:00 2024-03-18T15:47:39.409 INFO:teuthology.orchestra.run.smithi150.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 224 kB/s | 32 kB 00:00 2024-03-18T15:47:39.434 INFO:teuthology.orchestra.run.smithi191.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-03-18T15:47:39.493 INFO:teuthology.orchestra.run.smithi191.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-03-18T15:47:39.543 INFO:teuthology.orchestra.run.smithi191.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-18T15:47:39.593 INFO:teuthology.orchestra.run.smithi191.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 699 kB/s | 35 kB 00:00 2024-03-18T15:47:39.635 INFO:teuthology.orchestra.run.smithi191.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-18T15:47:39.702 INFO:teuthology.orchestra.run.smithi191.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.9 MB/s | 4.1 MB 00:00 2024-03-18T15:47:39.735 INFO:teuthology.orchestra.run.smithi191.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 936 kB/s | 94 kB 00:00 2024-03-18T15:47:39.836 INFO:teuthology.orchestra.run.smithi191.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-18T15:47:39.953 INFO:teuthology.orchestra.run.smithi191.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-03-18T15:47:39.995 INFO:teuthology.orchestra.run.smithi191.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-03-18T15:47:40.028 INFO:teuthology.orchestra.run.smithi191.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-03-18T15:47:40.078 INFO:teuthology.orchestra.run.smithi191.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-18T15:47:40.110 INFO:teuthology.orchestra.run.smithi150.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.0 MB/s | 747 kB 00:00 2024-03-18T15:47:40.112 INFO:teuthology.orchestra.run.smithi191.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 778 kB/s | 26 kB 00:00 2024-03-18T15:47:40.146 INFO:teuthology.orchestra.run.smithi191.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-18T15:47:40.212 INFO:teuthology.orchestra.run.smithi191.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-18T15:47:40.238 INFO:teuthology.orchestra.run.smithi191.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-18T15:47:40.271 INFO:teuthology.orchestra.run.smithi191.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-18T15:47:40.305 INFO:teuthology.orchestra.run.smithi191.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-18T15:47:40.318 INFO:teuthology.orchestra.run.smithi150.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 784 kB/s | 163 kB 00:00 2024-03-18T15:47:40.339 INFO:teuthology.orchestra.run.smithi191.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-03-18T15:47:40.356 INFO:teuthology.orchestra.run.smithi191.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 864 kB/s | 43 kB 00:00 2024-03-18T15:47:40.373 INFO:teuthology.orchestra.run.smithi191.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 952 kB/s | 32 kB 00:00 2024-03-18T15:47:40.398 INFO:teuthology.orchestra.run.smithi191.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 357 kB/s | 15 kB 00:00 2024-03-18T15:47:40.423 INFO:teuthology.orchestra.run.smithi191.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-18T15:47:40.449 INFO:teuthology.orchestra.run.smithi191.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-18T15:47:40.474 INFO:teuthology.orchestra.run.smithi191.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-18T15:47:40.499 INFO:teuthology.orchestra.run.smithi191.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 704 kB/s | 35 kB 00:00 2024-03-18T15:47:40.502 INFO:teuthology.orchestra.run.smithi150.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 692 kB/s | 127 kB 00:00 2024-03-18T15:47:40.525 INFO:teuthology.orchestra.run.smithi191.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-18T15:47:40.558 INFO:teuthology.orchestra.run.smithi191.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-03-18T15:47:40.608 INFO:teuthology.orchestra.run.smithi191.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-18T15:47:40.634 INFO:teuthology.orchestra.run.smithi191.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-18T15:47:40.659 INFO:teuthology.orchestra.run.smithi191.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-18T15:47:40.684 INFO:teuthology.orchestra.run.smithi191.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-18T15:47:40.709 INFO:teuthology.orchestra.run.smithi191.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-03-18T15:47:40.735 INFO:teuthology.orchestra.run.smithi191.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-18T15:47:40.768 INFO:teuthology.orchestra.run.smithi191.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-18T15:47:40.794 INFO:teuthology.orchestra.run.smithi191.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 792 kB/s | 46 kB 00:00 2024-03-18T15:47:40.819 INFO:teuthology.orchestra.run.smithi191.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-18T15:47:40.844 INFO:teuthology.orchestra.run.smithi191.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-18T15:47:40.869 INFO:teuthology.orchestra.run.smithi191.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-18T15:47:40.895 INFO:teuthology.orchestra.run.smithi191.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-03-18T15:47:40.920 INFO:teuthology.orchestra.run.smithi191.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-18T15:47:40.971 INFO:teuthology.orchestra.run.smithi191.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-18T15:47:40.994 INFO:teuthology.orchestra.run.smithi150.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 1.0 MB/s | 501 kB 00:00 2024-03-18T15:47:40.995 INFO:teuthology.orchestra.run.smithi191.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-03-18T15:47:41.021 INFO:teuthology.orchestra.run.smithi191.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-18T15:47:41.046 INFO:teuthology.orchestra.run.smithi191.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-18T15:47:41.071 INFO:teuthology.orchestra.run.smithi191.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-03-18T15:47:41.105 INFO:teuthology.orchestra.run.smithi191.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-03-18T15:47:41.130 INFO:teuthology.orchestra.run.smithi191.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-18T15:47:41.164 INFO:teuthology.orchestra.run.smithi191.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-18T15:47:41.189 INFO:teuthology.orchestra.run.smithi191.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-03-18T15:47:41.215 INFO:teuthology.orchestra.run.smithi191.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-03-18T15:47:41.240 INFO:teuthology.orchestra.run.smithi191.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-18T15:47:41.299 INFO:teuthology.orchestra.run.smithi191.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-18T15:47:42.715 INFO:teuthology.orchestra.run.smithi191.stdout:(150/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-03-18T15:47:42.882 INFO:teuthology.orchestra.run.smithi191.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.9 MB/s | 321 kB 00:00 2024-03-18T15:47:43.138 INFO:teuthology.orchestra.run.smithi156.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 1.9 MB/s | 9.8 MB 00:05 2024-03-18T15:47:43.158 INFO:teuthology.orchestra.run.smithi191.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 1.8 MB/s | 3.4 MB 00:01 2024-03-18T15:47:43.313 INFO:teuthology.orchestra.run.smithi156.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.2 MB/s | 221 kB 00:00 2024-03-18T15:47:43.396 INFO:teuthology.orchestra.run.smithi156.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 326 kB/s | 27 kB 00:00 2024-03-18T15:47:43.505 INFO:teuthology.orchestra.run.smithi156.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 292 kB/s | 32 kB 00:00 2024-03-18T15:47:43.516 INFO:teuthology.orchestra.run.smithi191.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.6 MB/s | 2.3 MB 00:00 2024-03-18T15:47:43.591 INFO:teuthology.orchestra.run.smithi191.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 567 kB/s | 42 kB 00:00 2024-03-18T15:47:43.717 INFO:teuthology.orchestra.run.smithi191.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-18T15:47:43.834 INFO:teuthology.orchestra.run.smithi191.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.7 MB/s | 1.8 MB 00:00 2024-03-18T15:47:43.859 INFO:teuthology.orchestra.run.smithi191.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 502 kB/s | 71 kB 00:00 2024-03-18T15:47:43.897 INFO:teuthology.orchestra.run.smithi156.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.9 MB/s | 747 kB 00:00 2024-03-18T15:47:43.943 INFO:teuthology.orchestra.run.smithi191.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 335 kB/s | 36 kB 00:00 2024-03-18T15:47:44.001 INFO:teuthology.orchestra.run.smithi191.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.3 MB/s | 192 kB 00:00 2024-03-18T15:47:44.052 INFO:teuthology.orchestra.run.smithi191.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-18T15:47:44.102 INFO:teuthology.orchestra.run.smithi191.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-18T15:47:44.106 INFO:teuthology.orchestra.run.smithi156.stdout:(15/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 2.4 MB/s | 17 MB 00:06 2024-03-18T15:47:44.131 INFO:teuthology.orchestra.run.smithi156.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 699 kB/s | 163 kB 00:00 2024-03-18T15:47:44.135 INFO:teuthology.orchestra.run.smithi191.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-18T15:47:44.194 INFO:teuthology.orchestra.run.smithi191.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-18T15:47:44.197 INFO:teuthology.orchestra.run.smithi156.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-18T15:47:44.253 INFO:teuthology.orchestra.run.smithi191.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-18T15:47:44.311 INFO:teuthology.orchestra.run.smithi191.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 546 kB/s | 64 kB 00:00 2024-03-18T15:47:44.336 INFO:teuthology.orchestra.run.smithi191.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 344 kB/s | 29 kB 00:00 2024-03-18T15:47:44.381 INFO:teuthology.orchestra.run.smithi156.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.0 MB/s | 501 kB 00:00 2024-03-18T15:47:44.481 INFO:teuthology.orchestra.run.smithi156.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 451 kB/s | 45 kB 00:00 2024-03-18T15:47:44.590 INFO:teuthology.orchestra.run.smithi156.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.2 MB/s | 131 kB 00:00 2024-03-18T15:47:44.674 INFO:teuthology.orchestra.run.smithi156.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-18T15:47:44.694 INFO:teuthology.orchestra.run.smithi191.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 3.2 MB/s | 1.2 MB 00:00 2024-03-18T15:47:44.829 INFO:teuthology.orchestra.run.smithi191.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 2.2 MB 00:00 2024-03-18T15:47:44.849 INFO:teuthology.orchestra.run.smithi156.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.8 MB/s | 320 kB 00:00 2024-03-18T15:47:44.936 INFO:teuthology.orchestra.run.smithi150.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 1.3 MB/s | 5.1 MB 00:03 2024-03-18T15:47:44.996 INFO:teuthology.orchestra.run.smithi191.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-03-18T15:47:45.016 INFO:teuthology.orchestra.run.smithi156.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 298 kB 00:00 2024-03-18T15:47:45.116 INFO:teuthology.orchestra.run.smithi156.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 997 kB/s | 99 kB 00:00 2024-03-18T15:47:45.129 INFO:teuthology.orchestra.run.smithi191.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 233 kB/s | 31 kB 00:00 2024-03-18T15:47:45.191 INFO:teuthology.orchestra.run.smithi156.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T15:47:45.228 INFO:teuthology.orchestra.run.smithi150.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 155 kB/s | 45 kB 00:00 2024-03-18T15:47:45.346 INFO:teuthology.orchestra.run.smithi191.stdout:(171/174): ceph-common-19.0.0-2162.g3d78f5ba.el 1.1 MB/s | 21 MB 00:18 2024-03-18T15:47:45.405 INFO:teuthology.orchestra.run.smithi191.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 2.3 MB 00:00 2024-03-18T15:47:45.420 INFO:teuthology.orchestra.run.smithi150.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 685 kB/s | 131 kB 00:00 2024-03-18T15:47:45.488 INFO:teuthology.orchestra.run.smithi191.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 526 kB/s | 74 kB 00:00 2024-03-18T15:47:45.597 INFO:teuthology.orchestra.run.smithi191.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.3 MB/s | 1.1 MB 00:00 2024-03-18T15:47:45.605 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:47:45.605 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 224 MB 00:19 2024-03-18T15:47:45.629 INFO:teuthology.orchestra.run.smithi150.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 776 kB/s | 162 kB 00:00 2024-03-18T15:47:45.912 INFO:teuthology.orchestra.run.smithi150.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.1 MB/s | 320 kB 00:00 2024-03-18T15:47:46.088 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-18T15:47:46.108 INFO:teuthology.orchestra.run.smithi156.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 2.7 MB/s | 5.1 MB 00:01 2024-03-18T15:47:46.179 INFO:teuthology.orchestra.run.smithi150.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 298 kB 00:00 2024-03-18T15:47:46.242 INFO:teuthology.orchestra.run.smithi156.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.3 MB/s | 170 kB 00:00 2024-03-18T15:47:46.304 INFO:teuthology.orchestra.run.smithi150.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 797 kB/s | 99 kB 00:00 2024-03-18T15:47:46.325 INFO:teuthology.orchestra.run.smithi156.stdout:(28/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 2.7 MB/s | 3.1 MB 00:01 2024-03-18T15:47:46.342 INFO:teuthology.orchestra.run.smithi156.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 338 kB/s | 34 kB 00:00 2024-03-18T15:47:46.360 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-18T15:47:46.360 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-18T15:47:46.418 INFO:teuthology.orchestra.run.smithi156.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-18T15:47:46.437 INFO:teuthology.orchestra.run.smithi150.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 640 kB/s | 85 kB 00:00 2024-03-18T15:47:47.235 INFO:teuthology.orchestra.run.smithi156.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 2.0 MB/s | 1.8 MB 00:00 2024-03-18T15:47:47.385 INFO:teuthology.orchestra.run.smithi156.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.6 MB/s | 248 kB 00:00 2024-03-18T15:47:47.443 INFO:teuthology.orchestra.run.smithi156.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 848 kB/s | 49 kB 00:00 2024-03-18T15:47:47.494 INFO:teuthology.orchestra.run.smithi156.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 329 kB/s | 16 kB 00:00 2024-03-18T15:47:47.652 INFO:teuthology.orchestra.run.smithi156.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.6 MB/s | 264 kB 00:00 2024-03-18T15:47:47.986 INFO:teuthology.orchestra.run.smithi156.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.2 MB/s | 748 kB 00:00 2024-03-18T15:47:48.045 INFO:teuthology.orchestra.run.smithi156.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 313 kB/s | 18 kB 00:00 2024-03-18T15:47:48.057 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-18T15:47:48.057 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-18T15:47:48.163 INFO:teuthology.orchestra.run.smithi150.stdout:(26/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 17 MB 00:12 2024-03-18T15:47:48.187 INFO:teuthology.orchestra.run.smithi156.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-18T15:47:48.213 INFO:teuthology.orchestra.run.smithi150.stdout:(27/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 1.7 MB/s | 3.1 MB 00:01 2024-03-18T15:47:48.238 INFO:teuthology.orchestra.run.smithi150.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-18T15:47:48.253 INFO:teuthology.orchestra.run.smithi156.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-18T15:47:48.263 INFO:teuthology.orchestra.run.smithi150.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 679 kB/s | 34 kB 00:00 2024-03-18T15:47:48.287 INFO:teuthology.orchestra.run.smithi156.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T15:47:48.297 INFO:teuthology.orchestra.run.smithi150.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-18T15:47:48.338 INFO:teuthology.orchestra.run.smithi156.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T15:47:48.372 INFO:teuthology.orchestra.run.smithi156.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-18T15:47:48.480 INFO:teuthology.orchestra.run.smithi156.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-18T15:47:48.513 INFO:teuthology.orchestra.run.smithi156.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 516 kB/s | 17 kB 00:00 2024-03-18T15:47:48.547 INFO:teuthology.orchestra.run.smithi156.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-18T15:47:48.589 INFO:teuthology.orchestra.run.smithi156.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T15:47:48.622 INFO:teuthology.orchestra.run.smithi156.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T15:47:48.672 INFO:teuthology.orchestra.run.smithi156.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T15:47:48.715 INFO:teuthology.orchestra.run.smithi156.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-18T15:47:48.748 INFO:teuthology.orchestra.run.smithi156.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-18T15:47:48.790 INFO:teuthology.orchestra.run.smithi156.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T15:47:48.841 INFO:teuthology.orchestra.run.smithi156.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T15:47:48.933 INFO:teuthology.orchestra.run.smithi156.stdout:(53/174): ceph-mgr-diskprediction-local-19.0.0- 2.9 MB/s | 7.4 MB 00:02 2024-03-18T15:47:48.958 INFO:teuthology.orchestra.run.smithi156.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-18T15:47:48.975 INFO:teuthology.orchestra.run.smithi156.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 824 kB/s | 34 kB 00:00 2024-03-18T15:47:48.989 INFO:teuthology.orchestra.run.smithi150.stdout:(31/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 4.3 MB/s | 48 MB 00:11 2024-03-18T15:47:49.056 INFO:teuthology.orchestra.run.smithi150.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 3.7 MB/s | 248 kB 00:00 2024-03-18T15:47:49.100 INFO:teuthology.orchestra.run.smithi156.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-18T15:47:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 983 kB/s | 49 kB 00:00 2024-03-18T15:47:49.133 INFO:teuthology.orchestra.run.smithi156.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 9.6 MB/s | 1.5 MB 00:00 2024-03-18T15:47:49.149 INFO:teuthology.orchestra.run.smithi150.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 395 kB/s | 16 kB 00:00 2024-03-18T15:47:49.176 INFO:teuthology.orchestra.run.smithi156.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-18T15:47:49.201 INFO:teuthology.orchestra.run.smithi156.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-18T15:47:49.215 INFO:teuthology.orchestra.run.smithi150.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 264 kB 00:00 2024-03-18T15:47:49.299 INFO:teuthology.orchestra.run.smithi150.stdout:(36/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 1.7 MB/s | 1.8 MB 00:01 2024-03-18T15:47:49.368 INFO:teuthology.orchestra.run.smithi156.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-18T15:47:49.374 INFO:teuthology.orchestra.run.smithi150.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-18T15:47:49.400 INFO:teuthology.orchestra.run.smithi150.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 182 kB/s | 18 kB 00:00 2024-03-18T15:47:49.402 INFO:teuthology.orchestra.run.smithi156.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-18T15:47:49.418 INFO:teuthology.orchestra.run.smithi156.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-18T15:47:49.596 INFO:teuthology.orchestra.run.smithi156.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-18T15:47:49.650 INFO:teuthology.orchestra.run.smithi150.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-18T15:47:49.683 INFO:teuthology.orchestra.run.smithi150.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 920 kB 00:00 2024-03-18T15:47:49.696 INFO:teuthology.orchestra.run.smithi156.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 88 kB/s | 24 kB 00:00 2024-03-18T15:47:49.709 INFO:teuthology.orchestra.run.smithi150.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-18T15:47:49.734 INFO:teuthology.orchestra.run.smithi150.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T15:47:49.759 INFO:teuthology.orchestra.run.smithi150.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-18T15:47:49.789 INFO:teuthology.orchestra.run.smithi156.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 94 kB/s | 18 kB 00:00 2024-03-18T15:47:49.814 INFO:teuthology.orchestra.run.smithi156.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-18T15:47:49.839 INFO:teuthology.orchestra.run.smithi156.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-18T15:47:49.876 INFO:teuthology.orchestra.run.smithi150.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-18T15:47:49.981 INFO:teuthology.orchestra.run.smithi156.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-18T15:47:49.985 INFO:teuthology.orchestra.run.smithi150.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 76 kB/s | 17 kB 00:00 2024-03-18T15:47:50.007 INFO:teuthology.orchestra.run.smithi156.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-18T15:47:50.032 INFO:teuthology.orchestra.run.smithi156.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T15:47:50.057 INFO:teuthology.orchestra.run.smithi156.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T15:47:50.068 INFO:teuthology.orchestra.run.smithi150.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 52 kB/s | 10 kB 00:00 2024-03-18T15:47:50.074 INFO:teuthology.orchestra.run.smithi156.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 984 kB/s | 41 kB 00:00 2024-03-18T15:47:50.086 INFO:teuthology.orchestra.run.smithi150.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.5 MB/s | 159 kB 00:00 2024-03-18T15:47:50.099 INFO:teuthology.orchestra.run.smithi156.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-18T15:47:50.102 INFO:teuthology.orchestra.run.smithi150.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T15:47:50.125 INFO:teuthology.orchestra.run.smithi156.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T15:47:50.136 INFO:teuthology.orchestra.run.smithi150.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T15:47:50.159 INFO:teuthology.orchestra.run.smithi156.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-18T15:47:50.161 INFO:teuthology.orchestra.run.smithi150.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-18T15:47:50.175 INFO:teuthology.orchestra.run.smithi156.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-18T15:47:50.178 INFO:teuthology.orchestra.run.smithi150.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-18T15:47:50.192 INFO:teuthology.orchestra.run.smithi156.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-18T15:47:50.203 INFO:teuthology.orchestra.run.smithi150.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T15:47:50.229 INFO:teuthology.orchestra.run.smithi150.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T15:47:50.254 INFO:teuthology.orchestra.run.smithi150.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-18T15:47:50.271 INFO:teuthology.orchestra.run.smithi150.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-03-18T15:47:50.355 INFO:teuthology.orchestra.run.smithi150.stdout:(56/174): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-03-18T15:47:50.426 INFO:teuthology.orchestra.run.smithi156.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 166 kB/s | 39 kB 00:00 2024-03-18T15:47:50.539 INFO:teuthology.orchestra.run.smithi150.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-18T15:47:50.572 INFO:teuthology.orchestra.run.smithi150.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.1 MB/s | 1.5 MB 00:00 2024-03-18T15:47:50.597 INFO:teuthology.orchestra.run.smithi150.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.3 MB/s | 811 kB 00:00 2024-03-18T15:47:50.622 INFO:teuthology.orchestra.run.smithi150.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.6 MB/s | 134 kB 00:00 2024-03-18T15:47:50.651 INFO:teuthology.orchestra.run.smithi156.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-18T15:47:50.668 INFO:teuthology.orchestra.run.smithi156.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 171 kB/s | 41 kB 00:00 2024-03-18T15:47:50.689 INFO:teuthology.orchestra.run.smithi150.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-03-18T15:47:50.693 INFO:teuthology.orchestra.run.smithi156.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-18T15:47:50.710 INFO:teuthology.orchestra.run.smithi156.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-18T15:47:50.727 INFO:teuthology.orchestra.run.smithi156.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T15:47:50.744 INFO:teuthology.orchestra.run.smithi156.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T15:47:50.769 INFO:teuthology.orchestra.run.smithi156.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-18T15:47:50.794 INFO:teuthology.orchestra.run.smithi156.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T15:47:50.806 INFO:teuthology.orchestra.run.smithi150.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-18T15:47:50.832 INFO:teuthology.orchestra.run.smithi150.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 374 kB/s | 78 kB 00:00 2024-03-18T15:47:50.845 INFO:teuthology.orchestra.run.smithi156.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-18T15:47:50.870 INFO:teuthology.orchestra.run.smithi156.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-18T15:47:50.896 INFO:teuthology.orchestra.run.smithi156.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T15:47:50.921 INFO:teuthology.orchestra.run.smithi156.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-18T15:47:50.924 INFO:teuthology.orchestra.run.smithi150.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-18T15:47:50.940 INFO:teuthology.orchestra.run.smithi150.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-18T15:47:50.957 INFO:teuthology.orchestra.run.smithi150.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-18T15:47:50.983 INFO:teuthology.orchestra.run.smithi150.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-03-18T15:47:51.008 INFO:teuthology.orchestra.run.smithi150.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-18T15:47:51.055 INFO:teuthology.orchestra.run.smithi156.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-18T15:47:51.105 INFO:teuthology.orchestra.run.smithi156.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-18T15:47:51.108 INFO:teuthology.orchestra.run.smithi150.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-18T15:47:51.133 INFO:teuthology.orchestra.run.smithi150.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-18T15:47:51.147 INFO:teuthology.orchestra.run.smithi156.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-18T15:47:51.158 INFO:teuthology.orchestra.run.smithi150.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-18T15:47:51.184 INFO:teuthology.orchestra.run.smithi150.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-18T15:47:51.209 INFO:teuthology.orchestra.run.smithi150.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-18T15:47:51.222 INFO:teuthology.orchestra.run.smithi156.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-18T15:47:51.234 INFO:teuthology.orchestra.run.smithi150.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-18T15:47:51.251 INFO:teuthology.orchestra.run.smithi150.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-18T15:47:51.255 INFO:teuthology.orchestra.run.smithi156.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-18T15:47:51.277 INFO:teuthology.orchestra.run.smithi150.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 5.0 MB/s | 211 kB 00:00 2024-03-18T15:47:51.302 INFO:teuthology.orchestra.run.smithi150.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 710 kB/s | 47 kB 00:00 2024-03-18T15:47:51.327 INFO:teuthology.orchestra.run.smithi150.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 571 kB/s | 43 kB 00:00 2024-03-18T15:47:51.423 INFO:teuthology.orchestra.run.smithi156.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 25 MB/s | 4.1 MB 00:00 2024-03-18T15:47:51.473 INFO:teuthology.orchestra.run.smithi156.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-18T15:47:51.523 INFO:teuthology.orchestra.run.smithi156.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-03-18T15:47:51.569 INFO:teuthology.orchestra.run.smithi150.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 162 kB/s | 39 kB 00:00 2024-03-18T15:47:51.582 INFO:teuthology.orchestra.run.smithi156.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-18T15:47:51.594 INFO:teuthology.orchestra.run.smithi150.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-03-18T15:47:51.624 INFO:teuthology.orchestra.run.smithi156.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 63 kB/s | 25 kB 00:00 2024-03-18T15:47:51.649 INFO:teuthology.orchestra.run.smithi156.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 521 kB/s | 35 kB 00:00 2024-03-18T15:47:51.691 INFO:teuthology.orchestra.run.smithi156.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 667 kB/s | 44 kB 00:00 2024-03-18T15:47:51.716 INFO:teuthology.orchestra.run.smithi156.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-18T15:47:51.745 INFO:teuthology.orchestra.run.smithi150.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-18T15:47:51.770 INFO:teuthology.orchestra.run.smithi150.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-18T15:47:51.795 INFO:teuthology.orchestra.run.smithi150.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 284 kB/s | 56 kB 00:00 2024-03-18T15:47:51.808 INFO:teuthology.orchestra.run.smithi156.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-18T15:47:51.820 INFO:teuthology.orchestra.run.smithi150.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2024-03-18T15:47:51.845 INFO:teuthology.orchestra.run.smithi150.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T15:47:51.871 INFO:teuthology.orchestra.run.smithi150.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-18T15:47:51.896 INFO:teuthology.orchestra.run.smithi150.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T15:47:51.917 INFO:teuthology.orchestra.run.smithi156.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-03-18T15:47:51.930 INFO:teuthology.orchestra.run.smithi150.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-18T15:47:51.947 INFO:teuthology.orchestra.run.smithi150.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-18T15:47:51.972 INFO:teuthology.orchestra.run.smithi150.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-18T15:47:51.997 INFO:teuthology.orchestra.run.smithi150.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 187 kB 00:00 2024-03-18T15:47:52.067 INFO:teuthology.orchestra.run.smithi156.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-03-18T15:47:52.092 INFO:teuthology.orchestra.run.smithi156.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-03-18T15:47:52.118 INFO:teuthology.orchestra.run.smithi156.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-03-18T15:47:52.160 INFO:teuthology.orchestra.run.smithi156.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-03-18T15:47:52.172 INFO:teuthology.orchestra.run.smithi150.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-18T15:47:52.197 INFO:teuthology.orchestra.run.smithi150.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 833 kB/s | 187 kB 00:00 2024-03-18T15:47:52.210 INFO:teuthology.orchestra.run.smithi156.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-03-18T15:47:52.231 INFO:teuthology.orchestra.run.smithi150.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-18T15:47:52.252 INFO:teuthology.orchestra.run.smithi156.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-18T15:47:52.265 INFO:teuthology.orchestra.run.smithi150.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-03-18T15:47:52.290 INFO:teuthology.orchestra.run.smithi150.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 232 kB/s | 21 kB 00:00 2024-03-18T15:47:52.310 INFO:teuthology.orchestra.run.smithi156.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-18T15:47:52.324 INFO:teuthology.orchestra.run.smithi150.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 275 kB/s | 25 kB 00:00 2024-03-18T15:47:52.382 INFO:teuthology.orchestra.run.smithi150.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-18T15:47:52.416 INFO:teuthology.orchestra.run.smithi150.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-18T15:47:52.441 INFO:teuthology.orchestra.run.smithi150.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-18T15:47:52.467 INFO:teuthology.orchestra.run.smithi150.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 694 kB/s | 35 kB 00:00 2024-03-18T15:47:52.492 INFO:teuthology.orchestra.run.smithi150.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 890 kB/s | 44 kB 00:00 2024-03-18T15:47:52.517 INFO:teuthology.orchestra.run.smithi150.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-18T15:47:52.544 INFO:teuthology.orchestra.run.smithi156.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-18T15:47:52.600 INFO:teuthology.orchestra.run.smithi150.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-18T15:47:52.659 INFO:teuthology.orchestra.run.smithi150.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-18T15:47:52.751 INFO:teuthology.orchestra.run.smithi150.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-18T15:47:52.802 INFO:teuthology.orchestra.run.smithi150.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.7 MB/s | 548 kB 00:00 2024-03-18T15:47:52.826 INFO:teuthology.orchestra.run.smithi150.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.8 MB/s | 309 kB 00:00 2024-03-18T15:47:52.844 INFO:teuthology.orchestra.run.smithi150.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-18T15:47:52.869 INFO:teuthology.orchestra.run.smithi150.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 620 kB/s | 26 kB 00:00 2024-03-18T15:47:52.869 INFO:teuthology.orchestra.run.smithi156.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-03-18T15:47:52.927 INFO:teuthology.orchestra.run.smithi150.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-18T15:47:52.928 INFO:teuthology.orchestra.run.smithi156.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 747 kB/s | 43 kB 00:00 2024-03-18T15:47:52.944 INFO:teuthology.orchestra.run.smithi150.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 483 kB/s | 49 kB 00:00 2024-03-18T15:47:52.986 INFO:teuthology.orchestra.run.smithi156.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-03-18T15:47:53.003 INFO:teuthology.orchestra.run.smithi150.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-18T15:47:53.037 INFO:teuthology.orchestra.run.smithi156.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-03-18T15:47:53.062 INFO:teuthology.orchestra.run.smithi150.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-18T15:47:53.112 INFO:teuthology.orchestra.run.smithi150.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-18T15:47:53.137 INFO:teuthology.orchestra.run.smithi156.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-18T15:47:53.137 INFO:teuthology.orchestra.run.smithi150.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 580 kB/s | 43 kB 00:00 2024-03-18T15:47:53.163 INFO:teuthology.orchestra.run.smithi150.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-18T15:47:53.188 INFO:teuthology.orchestra.run.smithi150.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-18T15:47:53.213 INFO:teuthology.orchestra.run.smithi150.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-18T15:47:53.247 INFO:teuthology.orchestra.run.smithi150.stdout:(120/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-18T15:47:53.280 INFO:teuthology.orchestra.run.smithi150.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-18T15:47:53.287 INFO:teuthology.orchestra.run.smithi156.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-03-18T15:47:53.322 INFO:teuthology.orchestra.run.smithi150.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 470 kB/s | 35 kB 00:00 2024-03-18T15:47:53.356 INFO:teuthology.orchestra.run.smithi150.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-18T15:47:53.371 INFO:teuthology.orchestra.run.smithi156.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-18T15:47:53.381 INFO:teuthology.orchestra.run.smithi150.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-18T15:47:53.405 INFO:teuthology.orchestra.run.smithi156.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 1.0 MB/s | 35 kB 00:00 2024-03-18T15:47:53.406 INFO:teuthology.orchestra.run.smithi150.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-18T15:47:53.432 INFO:teuthology.orchestra.run.smithi150.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-03-18T15:47:53.457 INFO:teuthology.orchestra.run.smithi150.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-18T15:47:53.472 INFO:teuthology.orchestra.run.smithi156.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-18T15:47:53.482 INFO:teuthology.orchestra.run.smithi150.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-18T15:47:53.508 INFO:teuthology.orchestra.run.smithi150.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-03-18T15:47:53.533 INFO:teuthology.orchestra.run.smithi150.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-03-18T15:47:53.592 INFO:teuthology.orchestra.run.smithi150.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-03-18T15:47:53.617 INFO:teuthology.orchestra.run.smithi150.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-03-18T15:47:53.642 INFO:teuthology.orchestra.run.smithi150.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-18T15:47:53.667 INFO:teuthology.orchestra.run.smithi150.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-18T15:47:53.699 INFO:teuthology.orchestra.run.smithi156.stdout:(123/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 4.4 MB 00:01 2024-03-18T15:47:53.701 INFO:teuthology.orchestra.run.smithi150.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-18T15:47:53.726 INFO:teuthology.orchestra.run.smithi150.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T15:47:53.751 INFO:teuthology.orchestra.run.smithi150.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-18T15:47:53.764 INFO:teuthology.orchestra.run.smithi156.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.2 MB/s | 144 kB 00:00 2024-03-18T15:47:53.777 INFO:teuthology.orchestra.run.smithi150.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-18T15:47:53.798 INFO:teuthology.orchestra.run.smithi156.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 322 kB/s | 11 kB 00:00 2024-03-18T15:47:53.802 INFO:teuthology.orchestra.run.smithi150.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 566 kB/s | 28 kB 00:00 2024-03-18T15:47:53.827 INFO:teuthology.orchestra.run.smithi150.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2024-03-18T15:47:53.831 INFO:teuthology.orchestra.run.smithi156.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 531 kB/s | 18 kB 00:00 2024-03-18T15:47:53.852 INFO:teuthology.orchestra.run.smithi150.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-18T15:47:53.865 INFO:teuthology.orchestra.run.smithi156.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 693 kB/s | 23 kB 00:00 2024-03-18T15:47:53.878 INFO:teuthology.orchestra.run.smithi150.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-03-18T15:47:53.903 INFO:teuthology.orchestra.run.smithi150.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-03-18T15:47:53.907 INFO:teuthology.orchestra.run.smithi156.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 468 kB/s | 19 kB 00:00 2024-03-18T15:47:53.928 INFO:teuthology.orchestra.run.smithi150.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-18T15:47:53.949 INFO:teuthology.orchestra.run.smithi156.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.3 MB/s | 2.0 MB 00:00 2024-03-18T15:47:53.954 INFO:teuthology.orchestra.run.smithi150.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-18T15:47:53.974 INFO:teuthology.orchestra.run.smithi156.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-03-18T15:47:53.979 INFO:teuthology.orchestra.run.smithi150.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-18T15:47:54.004 INFO:teuthology.orchestra.run.smithi150.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-03-18T15:47:54.029 INFO:teuthology.orchestra.run.smithi150.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-18T15:47:54.050 INFO:teuthology.orchestra.run.smithi156.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 619 kB/s | 46 kB 00:00 2024-03-18T15:47:54.054 INFO:teuthology.orchestra.run.smithi150.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-18T15:47:54.083 INFO:teuthology.orchestra.run.smithi156.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-03-18T15:47:54.097 INFO:teuthology.orchestra.run.smithi150.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-18T15:47:54.125 INFO:teuthology.orchestra.run.smithi156.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-03-18T15:47:54.208 INFO:teuthology.orchestra.run.smithi156.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 3.9 MB/s | 1.0 MB 00:00 2024-03-18T15:47:54.259 INFO:teuthology.orchestra.run.smithi156.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-03-18T15:47:54.284 INFO:teuthology.orchestra.run.smithi156.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-18T15:47:54.318 INFO:teuthology.orchestra.run.smithi156.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-18T15:47:54.330 INFO:teuthology.orchestra.run.smithi150.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.7 MB/s | 321 kB 00:00 2024-03-18T15:47:54.343 INFO:teuthology.orchestra.run.smithi156.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-18T15:47:54.368 INFO:teuthology.orchestra.run.smithi156.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-03-18T15:47:54.418 INFO:teuthology.orchestra.run.smithi156.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-18T15:47:54.452 INFO:teuthology.orchestra.run.smithi156.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-18T15:47:54.477 INFO:teuthology.orchestra.run.smithi156.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 610 kB/s | 36 kB 00:00 2024-03-18T15:47:54.502 INFO:teuthology.orchestra.run.smithi156.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-18T15:47:54.544 INFO:teuthology.orchestra.run.smithi156.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-18T15:47:54.570 INFO:teuthology.orchestra.run.smithi156.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-18T15:47:54.612 INFO:teuthology.orchestra.run.smithi156.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 480 kB/s | 20 kB 00:00 2024-03-18T15:47:54.679 INFO:teuthology.orchestra.run.smithi156.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-18T15:47:54.706 INFO:teuthology.orchestra.run.smithi150.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 6.1 MB/s | 2.3 MB 00:00 2024-03-18T15:47:54.720 INFO:teuthology.orchestra.run.smithi156.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-18T15:47:54.798 INFO:teuthology.orchestra.run.smithi150.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-03-18T15:47:54.964 INFO:teuthology.orchestra.run.smithi150.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 254 kB/s | 42 kB 00:00 2024-03-18T15:47:55.045 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-18T15:47:55.064 INFO:teuthology.orchestra.run.smithi150.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 5.1 MB/s | 1.8 MB 00:00 2024-03-18T15:47:55.123 INFO:teuthology.orchestra.run.smithi150.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-18T15:47:55.129 INFO:teuthology.orchestra.run.smithi156.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-03-18T15:47:55.140 INFO:teuthology.orchestra.run.smithi150.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 944 kB/s | 71 kB 00:00 2024-03-18T15:47:55.142 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:47:55.176 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:47:55.182 INFO:teuthology.orchestra.run.smithi150.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 623 kB/s | 36 kB 00:00 2024-03-18T15:47:55.207 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:47:55.207 INFO:teuthology.orchestra.run.smithi150.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-18T15:47:55.216 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:47:55.256 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:47:55.258 INFO:teuthology.orchestra.run.smithi150.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-03-18T15:47:55.283 INFO:teuthology.orchestra.run.smithi150.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-03-18T15:47:55.310 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:47:55.344 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:47:55.369 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:47:55.383 INFO:teuthology.orchestra.run.smithi150.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-18T15:47:55.412 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:47:55.692 INFO:teuthology.orchestra.run.smithi150.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 250 kB/s | 108 kB 00:00 2024-03-18T15:47:55.717 INFO:teuthology.orchestra.run.smithi150.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 406 kB/s | 136 kB 00:00 2024-03-18T15:47:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:47:55.743 INFO:teuthology.orchestra.run.smithi150.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-18T15:47:55.768 INFO:teuthology.orchestra.run.smithi150.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 571 kB/s | 29 kB 00:00 2024-03-18T15:47:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:47:55.901 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:47:55.930 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:47:55.977 INFO:teuthology.orchestra.run.smithi150.stdout:(167/174): librados2-19.0.0-2162.g3d78f5ba.el9. 1.7 MB/s | 3.4 MB 00:01 2024-03-18T15:47:56.021 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:47:56.051 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:47:56.098 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:47:56.110 INFO:teuthology.orchestra.run.smithi150.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.3 MB/s | 1.2 MB 00:00 2024-03-18T15:47:56.194 INFO:teuthology.orchestra.run.smithi150.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.1 MB/s | 2.2 MB 00:00 2024-03-18T15:47:56.246 INFO:teuthology.orchestra.run.smithi156.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.2 MB/s | 3.4 MB 00:01 2024-03-18T15:47:56.277 INFO:teuthology.orchestra.run.smithi150.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 374 kB/s | 31 kB 00:00 2024-03-18T15:47:56.282 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:47:56.331 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:47:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:47:56.463 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:47:56.486 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:47:56.488 INFO:teuthology.orchestra.run.smithi156.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.3 MB/s | 321 kB 00:00 2024-03-18T15:47:56.550 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:47:56.598 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:47:56.621 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:47:56.637 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:47:56.719 INFO:teuthology.orchestra.run.smithi150.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.4 MB/s | 1.1 MB 00:00 2024-03-18T15:47:56.763 INFO:teuthology.orchestra.run.smithi156.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.3 MB/s | 2.3 MB 00:00 2024-03-18T15:47:56.830 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:47:56.853 INFO:teuthology.orchestra.run.smithi150.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 558 kB/s | 74 kB 00:00 2024-03-18T15:47:56.860 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:47:56.892 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:47:56.897 INFO:teuthology.orchestra.run.smithi156.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-03-18T15:47:56.972 INFO:teuthology.orchestra.run.smithi156.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 565 kB/s | 42 kB 00:00 2024-03-18T15:47:56.995 INFO:teuthology.orchestra.run.smithi150.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.2 MB/s | 2.3 MB 00:01 2024-03-18T15:47:56.996 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:47:57.020 INFO:teuthology.orchestra.run.smithi150.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 511 kB/s | 465 kB 00:00 2024-03-18T15:47:57.022 INFO:teuthology.orchestra.run.smithi156.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-03-18T15:47:57.024 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:47:57.024 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.1 MB/s | 224 MB 00:27 2024-03-18T15:47:57.086 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:47:57.098 INFO:teuthology.orchestra.run.smithi156.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 951 kB/s | 71 kB 00:00 2024-03-18T15:47:57.146 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:47:57.219 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:47:57.240 INFO:teuthology.orchestra.run.smithi156.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 257 kB/s | 36 kB 00:00 2024-03-18T15:47:57.272 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:47:57.298 INFO:teuthology.orchestra.run.smithi156.stdout:(158/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-03-18T15:47:57.324 INFO:teuthology.orchestra.run.smithi156.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-18T15:47:57.382 INFO:teuthology.orchestra.run.smithi156.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 396 kB/s | 23 kB 00:00 2024-03-18T15:47:57.433 INFO:teuthology.orchestra.run.smithi156.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.9 MB/s | 256 kB 00:00 2024-03-18T15:47:57.458 INFO:teuthology.orchestra.run.smithi156.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-18T15:47:57.483 INFO:teuthology.orchestra.run.smithi156.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-18T15:47:57.489 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-18T15:47:57.493 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:47:57.508 INFO:teuthology.orchestra.run.smithi156.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-18T15:47:57.534 INFO:teuthology.orchestra.run.smithi156.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-18T15:47:57.576 INFO:teuthology.orchestra.run.smithi156.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 429 kB/s | 29 kB 00:00 2024-03-18T15:47:57.602 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:47:57.634 INFO:teuthology.orchestra.run.smithi156.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-03-18T15:47:57.698 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:47:57.727 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:47:57.754 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:47:57.784 INFO:teuthology.orchestra.run.smithi156.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.3 MB 00:00 2024-03-18T15:47:57.789 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:47:57.799 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-18T15:47:57.799 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-18T15:47:57.835 INFO:teuthology.orchestra.run.smithi156.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 9.1 MB/s | 465 kB 00:00 2024-03-18T15:47:57.863 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:47:57.910 INFO:teuthology.orchestra.run.smithi156.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.5 MB/s | 2.2 MB 00:00 2024-03-18T15:47:57.917 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:47:57.935 INFO:teuthology.orchestra.run.smithi156.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 310 kB/s | 31 kB 00:00 2024-03-18T15:47:57.964 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:47:57.977 INFO:teuthology.orchestra.run.smithi156.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.8 MB/s | 74 kB 00:00 2024-03-18T15:47:58.011 INFO:teuthology.orchestra.run.smithi156.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-03-18T15:47:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:47:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:47:58.107 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:47:58.140 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:47:58.168 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:47:58.239 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:47:58.274 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:47:58.316 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:47:58.345 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:47:58.423 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:47:58.477 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:47:58.551 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:47:58.703 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:47:58.754 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:47:59.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:47:59.376 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:47:59.458 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:47:59.503 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:47:59.511 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:47:59.543 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:47:59.600 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:47:59.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:47:59.676 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:47:59.793 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-18T15:47:59.793 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-18T15:47:59.929 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:48:00.084 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:48:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:48:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:48:00.345 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:48:00.401 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:48:00.464 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:48:00.538 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:48:00.568 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:48:00.605 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:48:00.652 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:48:00.688 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:48:00.731 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:48:00.774 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:48:00.916 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:48:01.046 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:48:01.076 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:48:01.117 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:48:01.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:48:01.259 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:48:01.299 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:48:01.324 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:48:01.349 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:48:01.430 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:48:01.559 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:48:01.665 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:48:01.707 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:48:01.756 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:48:01.811 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:48:02.007 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:48:02.062 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:48:02.199 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:48:02.235 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:48:02.275 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:48:02.314 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:48:02.343 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:48:02.376 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:48:02.416 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:48:02.447 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:48:02.477 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:48:02.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:48:02.588 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:48:02.622 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:48:02.663 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:48:02.705 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:48:02.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:48:02.893 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:48:02.937 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:48:02.976 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:48:03.023 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:48:03.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:48:03.169 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:48:03.907 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:48:03.958 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:48:03.992 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:48:04.040 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:48:04.138 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:48:04.176 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:48:04.178 INFO:teuthology.orchestra.run.smithi156.stdout:(174/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 1.8 MB/s | 48 MB 00:25 2024-03-18T15:48:04.182 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:48:04.183 INFO:teuthology.orchestra.run.smithi156.stdout:Total 6.8 MB/s | 224 MB 00:32 2024-03-18T15:48:04.354 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:48:04.388 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:48:04.469 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:48:04.660 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-18T15:48:04.930 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-18T15:48:04.930 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-18T15:48:05.110 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:48:05.224 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:48:06.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:48:06.535 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:48:06.566 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:48:06.597 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:48:06.624 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:48:06.648 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-18T15:48:06.648 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-18T15:48:06.905 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-18T15:48:06.946 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:48:06.992 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:48:07.000 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:07.017 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:48:07.079 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:48:07.088 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:48:07.094 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:07.126 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:48:07.135 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:48:07.162 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:48:07.195 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:48:07.233 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:48:07.267 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:48:07.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:48:07.621 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:48:07.649 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:07.653 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:48:07.709 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:07.742 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:48:07.821 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:07.852 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:07.943 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:07.975 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:48:08.023 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:48:08.200 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:48:08.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:48:08.282 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:08.368 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:08.391 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:48:08.457 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:48:08.504 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:48:08.527 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:48:08.542 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:48:08.759 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:48:08.794 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:48:08.841 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:48:09.003 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:48:09.135 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:48:09.196 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:48:09.312 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:09.380 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:09.599 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:09.710 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:48:09.805 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:48:09.836 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:48:09.862 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:48:09.906 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:48:09.971 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:48:10.025 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:48:10.064 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:48:10.137 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:48:10.172 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:48:10.215 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:48:10.247 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:48:10.275 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:48:10.345 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:48:10.388 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:48:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:48:10.461 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:48:10.496 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:10.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:48:10.539 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:10.575 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:48:10.650 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:48:10.786 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:48:10.837 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:48:11.077 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:11.125 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:11.190 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:11.190 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:48:11.190 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:48:11.190 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:11.219 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:11.366 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:48:11.449 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:48:11.531 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:48:11.585 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:48:11.593 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:48:11.627 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:48:11.684 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:48:11.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:48:11.753 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:48:12.027 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:48:12.211 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:48:12.279 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:48:12.323 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:48:12.460 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:48:12.527 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:48:12.582 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:48:12.655 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:48:12.686 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:48:12.714 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:48:12.762 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:48:12.796 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:48:12.840 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:48:12.883 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:48:13.021 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:48:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:48:13.100 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:48:13.143 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:48:13.237 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:48:13.284 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:48:13.324 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:48:13.350 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:48:13.374 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:48:13.455 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:48:13.550 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-18T15:48:13.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:48:13.658 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:48:13.692 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:48:13.693 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:48:13.723 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:48:13.732 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:48:13.741 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:48:13.772 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:48:13.799 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:48:13.826 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:48:13.859 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:48:13.860 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:48:13.894 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:48:13.935 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:48:14.051 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:48:14.095 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:48:14.236 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:48:14.257 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:48:14.277 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:48:14.308 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:48:14.320 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:14.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:48:14.367 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:48:14.402 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:48:14.423 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:14.441 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:48:14.454 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:14.474 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:48:14.502 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:48:14.528 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:14.558 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:48:14.590 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:48:14.599 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:48:14.621 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:48:14.654 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:48:14.696 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:48:14.731 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:48:14.765 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:48:14.820 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:48:14.823 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:48:14.843 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:14.911 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:48:14.929 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:14.952 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:48:14.953 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:48:14.993 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:48:14.999 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:48:15.039 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:48:15.040 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:48:15.062 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:48:15.078 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:48:15.148 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:48:15.228 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:48:15.237 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:48:15.268 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:48:15.300 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:48:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:48:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:48:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:48:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:15.723 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:15.931 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:48:15.965 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:48:16.016 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:48:16.034 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:48:16.050 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:48:16.099 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:48:16.131 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:48:16.169 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:48:16.188 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:48:16.196 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:48:16.225 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:48:16.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:48:16.305 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:48:16.367 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:48:16.403 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:48:16.406 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:48:16.440 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:48:16.469 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:48:16.506 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:48:16.518 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:48:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:48:16.581 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:48:16.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:48:16.678 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:48:16.715 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:48:16.764 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:48:16.939 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:48:17.130 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:48:17.173 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:48:17.176 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:48:17.250 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:48:17.383 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:48:17.429 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:48:17.481 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:48:17.947 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:48:18.025 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:48:18.107 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:48:18.145 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:48:18.153 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:48:18.185 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:48:18.234 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:48:18.272 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:48:18.310 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:48:18.549 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:48:18.719 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:48:18.736 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:48:18.780 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:48:18.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:48:18.832 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:48:18.855 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:48:18.886 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:48:18.914 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:48:18.982 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:48:19.027 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:48:19.082 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:48:19.163 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:48:19.194 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:48:19.223 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:48:19.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:48:19.252 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:19.278 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:48:19.310 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:48:19.342 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:19.353 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:48:19.373 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:48:19.395 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:48:19.410 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:48:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:48:19.602 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:48:19.632 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:48:19.673 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:48:19.767 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:48:19.807 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:48:19.847 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:48:19.867 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:48:19.873 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:48:19.897 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:48:19.905 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:19.959 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:19.969 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:48:19.981 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:48:20.091 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:48:20.190 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:48:20.239 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:48:20.287 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:48:20.343 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:48:20.526 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:48:20.568 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:48:20.698 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:48:20.742 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:48:20.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:48:20.805 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:48:20.833 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:48:20.867 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:48:20.906 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:48:20.938 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:48:20.968 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:48:21.043 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:48:21.078 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:48:21.103 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:48:21.144 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:48:21.171 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:48:21.260 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:48:21.350 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:48:21.394 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:48:21.425 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:48:21.472 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:48:21.578 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:48:21.635 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:48:22.280 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:22.321 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:48:22.408 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:48:22.441 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:48:22.489 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:48:22.568 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:48:22.609 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:48:22.785 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:48:22.798 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-18T15:48:22.799 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:22.824 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:48:22.859 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:22.891 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:48:22.964 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:22.981 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:22.981 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:22.981 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:48:22.981 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:22.984 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:22.984 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:48:22.984 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:48:22.984 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:23.026 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:23.540 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:48:23.655 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:48:24.459 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:24.485 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:24.718 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:24.741 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:24.741 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:24.741 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:48:24.741 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:24.741 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:24.742 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:48:24.967 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:48:24.998 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:48:25.029 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:48:25.057 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:48:25.194 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:25.217 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:25.304 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:48:25.368 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:48:25.404 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:25.494 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:48:25.525 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:48:25.562 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:48:25.696 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:25.727 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:25.768 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:48:25.827 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:48:25.859 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:26.019 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:48:26.057 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:26.097 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:26.127 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:48:26.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:26.150 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:48:27.105 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:27.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:27.235 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:27.398 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:48:27.429 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:27.452 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:27.452 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:27.452 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:48:27.452 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:27.453 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:27.453 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:27.496 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:27.507 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:27.552 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:48:28.375 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:28.414 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:28.430 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:48:28.457 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:28.479 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:28.479 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:28.480 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:48:28.480 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:28.480 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:28.480 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:28.524 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:28.546 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:28.547 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:28.547 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:48:28.547 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:28.600 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:28.622 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:28.622 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:28.623 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:48:28.623 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:28.623 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:28.623 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:28.949 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:28.965 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:28.984 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:29.066 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:48:29.066 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:48:29.066 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:48:29.066 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:29.093 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:33.405 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:48:33.446 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:48:33.477 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:48:33.512 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:48:33.693 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:48:33.821 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:48:33.860 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:48:33.899 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:48:33.972 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:48:34.012 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:48:34.044 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:48:34.085 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:48:34.114 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:48:34.175 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:48:34.230 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:48:34.271 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:48:34.619 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:34.619 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-03-18T15:48:34.619 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-03-18T15:48:34.619 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-03-18T15:48:34.620 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-03-18T15:48:34.620 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-03-18T15:48:34.620 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-03-18T15:48:34.620 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-03-18T15:48:34.620 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:34.788 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:34.803 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:34.803 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:34.803 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:48:34.803 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:34.884 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:34.919 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:34.939 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:35.764 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:48:35.856 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:48:35.897 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:48:35.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:48:36.010 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:48:36.035 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:48:36.068 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:48:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:48:36.095 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:36.182 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:36.212 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:48:36.212 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:36.274 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:36.297 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:36.299 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:36.299 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:36.299 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:48:36.300 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:36.300 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:36.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:36.331 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:48:36.365 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:48:36.390 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:48:36.415 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:48:36.448 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:48:36.487 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:48:36.520 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:48:36.525 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:36.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:36.553 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:48:36.587 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:48:36.612 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:48:36.638 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:48:36.645 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:36.645 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:36.824 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:36.824 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:36.915 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:48:36.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:37.007 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:48:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:48:37.102 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:48:37.117 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:48:37.503 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:37.534 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:37.566 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:48:37.627 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:48:37.657 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:37.892 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:37.924 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:38.887 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:38.908 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:39.010 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:39.170 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:48:39.202 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:39.225 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:39.269 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:39.280 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:39.332 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:48:40.199 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:48:40.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:40.297 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:40.318 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:40.318 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:40.318 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:48:40.319 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:40.364 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:40.386 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:40.386 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:40.386 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:48:40.386 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:40.386 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:40.387 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:40.698 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:40.720 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:40.721 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:40.721 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:48:40.721 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:40.721 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:40.721 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:41.041 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:48:41.041 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-03-18T15:48:41.041 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-03-18T15:48:41.042 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:41.201 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:41.217 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:48:41.217 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:41.217 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:48:41.217 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:42.332 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:42.333 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:48:42.334 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:48:42.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:48:42.336 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:48:42.337 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:48:42.338 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:48:42.339 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:48:42.340 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:48:42.341 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:48:42.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:48:42.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:48:42.344 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:48:42.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:48:42.346 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:48:42.713 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:42.738 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:48:42.738 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:42.739 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:48:42.739 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:42.739 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:48:42.739 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:42.963 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:48:42.986 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:43.422 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:48:43.445 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:43.541 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:48:43.975 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:44.006 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:48:44.038 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:48:44.089 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:48:44.113 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:44.344 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:48:44.380 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:45.177 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:48:45.224 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:48:45.257 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:48:45.285 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:48:45.341 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:48:45.364 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:45.408 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:48:45.408 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:45.408 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-18T15:48:45.408 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:45.408 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.409 INFO:teuthology.orchestra.run.smithi191.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:48:45.410 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.411 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:48:45.412 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.413 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:48:45.414 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:48:45.415 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:48:45.416 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:48:45.417 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:48:45.418 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:48:45.419 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.420 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:48:45.421 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-18T15:48:45.464 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:48:45.470 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:48:45.592 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:48:45.617 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:48:45.633 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:48:45.651 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:48:45.675 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:45.688 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:48:45.719 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:45.730 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:48:45.757 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:48:45.775 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:48:45.800 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:48:45.853 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:48:45.891 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:48:45.920 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:48:45.926 DEBUG:teuthology.parallel:result is None 2024-03-18T15:48:46.004 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:48:46.069 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:48:46.094 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:48:46.642 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:48:46.671 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:46.693 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:48:46.694 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:46.694 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:48:46.694 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:46.694 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:48:46.694 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:46.710 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:46.738 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:46.750 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:48:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:46.760 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:48:46.760 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:46.770 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:46.806 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:48:46.828 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:47.140 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:48:47.163 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:48:47.622 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:48:47.703 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:48:47.744 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:48:47.827 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:48:47.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:48:47.890 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:48:47.915 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:48:47.941 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:48:47.942 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:48.013 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:48.043 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:48:48.043 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:48.111 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:48.137 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:48:48.178 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:48:48.204 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:48:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:48:48.263 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:48:48.309 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:48:48.350 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:48:48.375 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:48:48.428 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:48:48.453 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:48:48.459 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:48.459 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:48.639 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:48.639 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:48.722 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:48.746 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:48:48.772 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:48:48.805 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:48:48.839 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:48:48.905 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:48:51.578 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:48:51.627 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:48:51.657 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:48:51.685 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:48:51.864 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:48:51.976 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:48:52.016 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:48:52.055 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:48:52.154 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:48:52.193 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:48:52.217 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:48:52.258 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:48:52.288 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:48:52.333 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:48:52.387 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:48:52.411 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:48:52.990 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:53.027 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:53.047 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:48:53.672 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:48:53.704 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:48:53.705 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:48:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:48:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:48:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:48:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:48:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:48:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:48:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:48:53.848 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:48:53.849 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:48:53.850 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:48:53.851 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:48:53.851 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:48:53.851 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:48:53.851 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:48:53.852 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:48:53.853 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:48:53.853 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:48:53.853 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:48:53.853 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:48:53.854 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:48:53.854 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:48:53.854 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:48:53.855 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:48:53.855 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:48:53.856 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:48:53.857 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:48:53.858 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:48:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:48:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:48:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:48:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:48:53.859 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:48:53.886 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:48:53.964 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:48:53.996 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:48:54.084 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:48:54.126 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:48:54.151 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:48:54.176 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:48:54.203 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:48:54.203 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:54.291 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:48:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:48:54.321 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:54.397 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:48:54.423 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:48:54.457 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:48:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:48:54.507 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:48:54.540 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:48:54.580 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:48:54.612 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:48:54.637 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:48:54.663 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:48:54.688 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:48:54.713 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:48:54.720 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:54.720 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:48:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:54.982 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:48:55.007 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:48:55.033 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:48:55.065 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:48:55.092 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:48:55.156 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:48:56.696 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:56.697 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:48:56.698 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.699 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.700 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:48:56.701 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:48:56.702 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:48:56.703 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:48:56.704 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:48:56.705 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:48:56.706 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:48:56.707 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.708 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:48:56.709 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:48:56.710 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:48:56.711 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:48:56.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:48:56.711 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-18T15:48:57.157 DEBUG:teuthology.parallel:result is None 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:49:00.191 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:49:00.192 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:49:00.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:49:00.208 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:49:00.209 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:49:00.210 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:49:00.211 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:49:00.212 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:49:00.213 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:49:00.214 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:49:00.215 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:49:00.216 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:49:00.217 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:49:00.218 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:49:00.219 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:49:00.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:49:03.311 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:49:03.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:49:03.311 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.312 INFO:teuthology.orchestra.run.smithi156.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-18T15:49:03.313 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.314 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.315 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.316 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:49:03.317 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:49:03.318 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:49:03.319 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:49:03.320 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:49:03.321 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:49:03.322 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:49:03.323 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:49:03.324 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-18T15:49:03.325 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-18T15:49:03.820 DEBUG:teuthology.parallel:result is None 2024-03-18T15:49:03.820 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:04.039 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:49:04.065 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.066 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.066 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:49:04.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:04.284 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:49:04.307 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.307 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.307 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:49:04.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:04.450 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:49:04.476 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.477 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:49:04.477 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:49:04.479 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T15:49:04.479 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:49:04.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:49:04.514 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:49:04.514 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:49:04.547 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:49:04.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:49:04.582 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T15:49:04.582 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:49:04.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:49:04.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:49:04.682 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:49:04.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:49:04.712 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:49:04.780 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:49:04.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:49:04.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:49:04.884 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T15:49:04.885 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:49:04.885 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:49:04.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:49:04.985 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:49:04.985 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:49:05.014 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:49:05.082 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:49:05.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:49:05.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:49:05.177 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T15:49:05.178 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:49:05.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:49:05.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:49:05.278 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:49:05.278 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:49:05.309 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:49:05.378 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:49:05.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:49:05.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:49:05.474 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T15:49:05.572 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-18T15:49:05.573 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:05.573 INFO:tasks.cephadm:Cluster fsid is 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:49:05.573 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T15:49:05.573 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.150', 'mon.b': '172.21.15.156', 'mon.c': '172.21.15.191'} 2024-03-18T15:49:05.574 INFO:tasks.cephadm:First mon is mon.a on smithi150 2024-03-18T15:49:05.574 INFO:tasks.cephadm:First mgr is x 2024-03-18T15:49:05.574 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T15:49:05.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-03-18T15:49:05.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-03-18T15:49:05.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-03-18T15:49:05.663 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T15:49:05.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:05.808 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T15:49:06.051 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:49:06.053 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T15:49:06.053 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:06.693 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:49 /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:06.693 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.130 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:49 /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.131 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.574 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:49 /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.575 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.592 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.611 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:49:07.645 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T15:49:07.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:49:07.648 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:49:07.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:49:08.490 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:49:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:49:08.618 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:49:45.588 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:49:45.588 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:49:45.588 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:49:45.589 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-03-18T15:49:45.589 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:49:45.589 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T15:49:45.589 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:49:48.934 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-03-18T15:49:48.935 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-03-18T15:49:50.363 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T15:49:50.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-03-18T15:49:50.421 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-03-18T15:49:50.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-03-18T15:49:50.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-03-18T15:49:50.517 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-03-18T15:49:50.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-03-18T15:49:50.573 INFO:tasks.cephadm:Writing seed config... 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T15:49:50.574 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-18T15:49:50.575 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T15:49:50.576 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T15:49:50.577 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-18T15:49:50.578 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:49:50.578 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T15:49:50.596 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0cd032e0-e53f-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T15:49:50.596 DEBUG:teuthology.orchestra.run.smithi150:mon.a> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T15:49:50.638 DEBUG:teuthology.orchestra.run.smithi150:mgr.x> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T15:49:50.681 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T15:49:50.681 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.150 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:49:50.876 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:49:50.877 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', '0cd032e0-e53f-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.150', '--skip-admin-label'] 2024-03-18T15:49:50.901 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:49:50.901 INFO:teuthology.orchestra.run.smithi150.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T15:49:50.901 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying podman|docker is present... 2024-03-18T15:49:50.925 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:49:50.925 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying lvm2 is present... 2024-03-18T15:49:50.926 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying time synchronization is in place... 2024-03-18T15:49:50.933 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:49:50.933 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:49:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:49:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:49:50.945 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout enabled 2024-03-18T15:49:50.951 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout active 2024-03-18T15:49:50.951 INFO:teuthology.orchestra.run.smithi150.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:49:50.952 INFO:teuthology.orchestra.run.smithi150.stdout:Repeating the final host check... 2024-03-18T15:49:50.975 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:49:50.975 INFO:teuthology.orchestra.run.smithi150.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T15:49:50.975 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl is present 2024-03-18T15:49:50.975 INFO:teuthology.orchestra.run.smithi150.stdout:lvcreate is present 2024-03-18T15:49:50.981 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:49:50.981 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:49:50.987 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:49:50.987 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:49:50.993 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout enabled 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout active 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:Host looks OK 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:Cluster fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:Acquiring lock 139625116696000 on /run/cephadm/0cd032e0-e53f-11ee-95c9-87774f69a715.lock 2024-03-18T15:49:50.999 INFO:teuthology.orchestra.run.smithi150.stdout:Lock 139625116696000 acquired on /run/cephadm/0cd032e0-e53f-11ee-95c9-87774f69a715.lock 2024-03-18T15:49:51.000 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying IP 172.21.15.150 port 3300 ... 2024-03-18T15:49:51.000 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying IP 172.21.15.150 port 6789 ... 2024-03-18T15:49:51.000 INFO:teuthology.orchestra.run.smithi150.stdout:Base mon IP(s) is [172.21.15.150:3300, 172.21.15.150:6789], mon addrv is [v2:172.21.15.150:3300,v1:172.21.15.150:6789] 2024-03-18T15:49:51.003 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.150 metric 100 2024-03-18T15:49:51.003 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.150 metric 100 2024-03-18T15:49:51.006 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T15:49:51.006 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-18T15:49:51.006 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1677sec hoplimit 64 pref medium 2024-03-18T15:49:51.008 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T15:49:51.008 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T15:49:51.008 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:Mon IP `172.21.15.150` is in CIDR network `172.21.0.0/20` 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:Mon IP `172.21.15.150` is in CIDR network `172.21.0.0/20` 2024-03-18T15:49:51.009 INFO:teuthology.orchestra.run.smithi150.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T15:49:51.010 INFO:teuthology.orchestra.run.smithi150.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T15:49:51.010 INFO:teuthology.orchestra.run.smithi150.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:49:51.439 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:49:51.440 INFO:teuthology.orchestra.run.smithi150.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T15:49:52.353 INFO:teuthology.orchestra.run.smithi150.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:49:52.353 INFO:teuthology.orchestra.run.smithi150.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:49:52.353 INFO:teuthology.orchestra.run.smithi150.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T15:49:53.367 INFO:teuthology.orchestra.run.smithi150.stdout:stat: stdout 167 167 2024-03-18T15:49:53.367 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial keys... 2024-03-18T15:49:54.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQChYvhlfDNaJxAA/sVGPqNGs2C5K6DlUVYERQ== 2024-03-18T15:49:55.173 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQCiYvhlN+HCIhAAWs8KkRMj82Jwl0Oso1oNlg== 2024-03-18T15:49:56.068 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph-authtool: stdout AQCjYvhl2I8dHRAAyWhxuRpj5+8CjjiNJEffBA== 2024-03-18T15:49:56.069 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial monmap... 2024-03-18T15:49:57.015 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool for a [v2:172.21.15.150:3300,v1:172.21.15.150:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = quincy 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: set fsid to 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:49:57.016 INFO:teuthology.orchestra.run.smithi150.stdout:Creating mon... 2024-03-18T15:49:57.888 INFO:teuthology.orchestra.run.smithi150.stdout:create mon.a on 2024-03-18T15:49:58.087 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T15:49:58.226 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T15:49:58.416 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target. 2024-03-18T15:49:58.416 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target. 2024-03-18T15:49:58.636 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a 2024-03-18T15:49:58.636 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T15:49:58.799 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target.wants/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@.service. 2024-03-18T15:49:59.346 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:49:59.346 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-03-18T15:49:59.353 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:49:59.353 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:49:59.353 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-03-18T15:49:59.353 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:49:59.354 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mon to start... 2024-03-18T15:49:59.354 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mon... 2024-03-18T15:50:00.505 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:00 smithi150 ceph-mon[31565]: from='client.? 172.21.15.150:0/45757918' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T15:50:00.507 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout id: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout services: 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.55472s) 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:00.513 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout data: 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.794+0000 7fd488154700 1 Processor -- start 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.794+0000 7fd488154700 1 -- start start 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd488154700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd488154700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4801075d0 con 0x7fd480106be0 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd485ef0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd485ef0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60712/0 (socket says 172.21.15.150:60712) 2024-03-18T15:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd485ef0700 1 -- 172.21.15.150:0/567601556 learned_addr learned my addr 172.21.15.150:0/567601556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.795+0000 7fd485ef0700 1 -- 172.21.15.150:0/567601556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd480107de0 con 0x7fd480106be0 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd485ef0700 1 --2- 172.21.15.150:0/567601556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd470009ab0 tx=0x7fd470009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1512fee54fed761c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd484eee700 1 -- 172.21.15.150:0/567601556 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd470004160 con 0x7fd480106be0 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd484eee700 1 -- 172.21.15.150:0/567601556 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd470004300 con 0x7fd480106be0 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd484eee700 1 -- 172.21.15.150:0/567601556 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd47000b7e0 con 0x7fd480106be0 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 -- 172.21.15.150:0/567601556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 msgr2=0x7fd480107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 --2- 172.21.15.150:0/567601556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd470009ab0 tx=0x7fd470009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 -- 172.21.15.150:0/567601556 shutdown_connections 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 --2- 172.21.15.150:0/567601556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd480107000 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 -- 172.21.15.150:0/567601556 >> 172.21.15.150:0/567601556 conn(0x7fd4801021c0 msgr2=0x7fd4801045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.796+0000 7fd488154700 1 -- 172.21.15.150:0/567601556 shutdown_connections 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd488154700 1 -- 172.21.15.150:0/567601556 wait complete. 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd488154700 1 Processor -- start 2024-03-18T15:50:00.515 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd488154700 1 -- start start 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd488154700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd488154700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd480108310 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd485ef0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd485ef0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60716/0 (socket says 172.21.15.150:60716) 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.797+0000 7fd485ef0700 1 -- 172.21.15.150:0/45757918 learned_addr learned my addr 172.21.15.150:0/45757918 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd485ef0700 1 -- 172.21.15.150:0/45757918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48019bf50 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd485ef0700 1 --2- 172.21.15.150:0/45757918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd470007fd0 tx=0x7fd470004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4700036a0 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47001f070 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4801a0560 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4700039b0 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4801a0900 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd470024470 con 0x7fd480106be0 2024-03-18T15:50:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.798+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd47001a850 con 0x7fd480106be0 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.799+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4800693d0 con 0x7fd480106be0 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.801+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd47001aaf0 con 0x7fd480106be0 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.873+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fd48002de30 con 0x7fd480106be0 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.873+0000 7fd476ffd700 1 -- 172.21.15.150:0/45757918 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7fd47001acd0 con 0x7fd480106be0 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 msgr2=0x7fd48007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 --2- 172.21.15.150:0/45757918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd470007fd0 tx=0x7fd470004750 comp rx=0 tx=0).stop 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 shutdown_connections 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 --2- 172.21.15.150:0/45757918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd480106be0 0x7fd48007c2d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 >> 172.21.15.150:0/45757918 conn(0x7fd4801021c0 msgr2=0x7fd480190f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 shutdown_connections 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:49:59.874+0000 7fd488154700 1 -- 172.21.15.150:0/45757918 wait complete. 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:mon is available 2024-03-18T15:50:00.517 INFO:teuthology.orchestra.run.smithi150.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T15:50:01.588 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:01 smithi150 ceph-mon[31565]: from='client.? 172.21.15.150:0/553260563' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:50:01.589 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:01 smithi150 ceph-mon[31565]: from='client.? 172.21.15.150:0/553260563' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout fsid = 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.150:3300,v1:172.21.15.150:6789] 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:50:01.597 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.931+0000 7fad90e68700 1 Processor -- start 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.931+0000 7fad90e68700 1 -- start start 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.931+0000 7fad90e68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.931+0000 7fad90e68700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c1086e0 con 0x7fad8c107cf0 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.931+0000 7fad8a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60720/0 (socket says 172.21.15.150:60720) 2024-03-18T15:50:01.598 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8a59c700 1 -- 172.21.15.150:0/4039458208 learned_addr learned my addr 172.21.15.150:0/4039458208 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8a59c700 1 -- 172.21.15.150:0/4039458208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8c108ef0 con 0x7fad8c107cf0 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8a59c700 1 --2- 172.21.15.150:0/4039458208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009ab0 tx=0x7fad7c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2194f78eefbe206f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8959a700 1 -- 172.21.15.150:0/4039458208 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad7c004160 con 0x7fad8c107cf0 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8959a700 1 -- 172.21.15.150:0/4039458208 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7c004300 con 0x7fad8c107cf0 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.932+0000 7fad8959a700 1 -- 172.21.15.150:0/4039458208 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad7c00b7e0 con 0x7fad8c107cf0 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 -- 172.21.15.150:0/4039458208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 msgr2=0x7fad8c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 --2- 172.21.15.150:0/4039458208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009ab0 tx=0x7fad7c009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 -- 172.21.15.150:0/4039458208 shutdown_connections 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 --2- 172.21.15.150:0/4039458208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c108110 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 -- 172.21.15.150:0/4039458208 >> 172.21.15.150:0/4039458208 conn(0x7fad8c1032b0 msgr2=0x7fad8c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 -- 172.21.15.150:0/4039458208 shutdown_connections 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 -- 172.21.15.150:0/4039458208 wait complete. 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.933+0000 7fad90e68700 1 Processor -- start 2024-03-18T15:50:01.599 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad90e68700 1 -- start start 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad90e68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad90e68700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c109420 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad8a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad8a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60726/0 (socket says 172.21.15.150:60726) 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad8a59c700 1 -- 172.21.15.150:0/553260563 learned_addr learned my addr 172.21.15.150:0/553260563 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad8a59c700 1 -- 172.21.15.150:0/553260563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8c19b960 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad8a59c700 1 --2- 172.21.15.150:0/553260563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fad7c007fd0 tx=0x7fad7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad7c0036a0 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7c01f070 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8c19bbf0 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad7c0039b0 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.934+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8c19e8e0 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.935+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fad7c024470 con 0x7fad8c107cf0 2024-03-18T15:50:01.600 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.935+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fad7c01a850 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.935+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad8c0693d0 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:00.938+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fad7c032020 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.006+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fad8c02dcf0 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.009+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7fad7c01aa50 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.009+0000 7fad7b7fe700 1 -- 172.21.15.150:0/553260563 <== mon.0 v2:172.21.15.150:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fad7c024d60 con 0x7fad8c107cf0 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 msgr2=0x7fad8c19b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 --2- 172.21.15.150:0/553260563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fad7c007fd0 tx=0x7fad7c004750 comp rx=0 tx=0).stop 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 shutdown_connections 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 --2- 172.21.15.150:0/553260563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad8c107cf0 0x7fad8c19b420 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 >> 172.21.15.150:0/553260563 conn(0x7fad8c1032b0 msgr2=0x7fad8c192040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 shutdown_connections 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.010+0000 7fad90e68700 1 -- 172.21.15.150:0/553260563 wait complete. 2024-03-18T15:50:01.601 INFO:teuthology.orchestra.run.smithi150.stdout:Generating new minimal ceph.conf... 2024-03-18T15:50:02.537 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:02 smithi150 ceph-mon[31565]: from='client.? 172.21.15.150:0/186555766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:50:02.547 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc62196e700 1 Processor -- start 2024-03-18T15:50:02.547 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc62196e700 1 -- start start 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc62196e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc62196e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1086e0 con 0x7fc61c107cf0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc61affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc61affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60742/0 (socket says 172.21.15.150:60742) 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.939+0000 7fc61affd700 1 -- 172.21.15.150:0/1087344171 learned_addr learned my addr 172.21.15.150:0/1087344171 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.940+0000 7fc61affd700 1 -- 172.21.15.150:0/1087344171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c108ef0 con 0x7fc61c107cf0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.940+0000 7fc61affd700 1 --2- 172.21.15.150:0/1087344171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009ab0 tx=0x7fc60c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c39a042c5c3266 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.940+0000 7fc619ffb700 1 -- 172.21.15.150:0/1087344171 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc60c004160 con 0x7fc61c107cf0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.940+0000 7fc619ffb700 1 -- 172.21.15.150:0/1087344171 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc60c00bca0 con 0x7fc61c107cf0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.940+0000 7fc619ffb700 1 -- 172.21.15.150:0/1087344171 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc60c003d60 con 0x7fc61c107cf0 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- 172.21.15.150:0/1087344171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 msgr2=0x7fc61c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 --2- 172.21.15.150:0/1087344171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009ab0 tx=0x7fc60c009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- 172.21.15.150:0/1087344171 shutdown_connections 2024-03-18T15:50:02.548 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 --2- 172.21.15.150:0/1087344171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c108110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- 172.21.15.150:0/1087344171 >> 172.21.15.150:0/1087344171 conn(0x7fc61c1032b0 msgr2=0x7fc61c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- 172.21.15.150:0/1087344171 shutdown_connections 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- 172.21.15.150:0/1087344171 wait complete. 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 Processor -- start 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.941+0000 7fc62196e700 1 -- start start 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62196e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62196e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c10c2a0 con 0x7fc61c107cf0 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc61affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc61affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60754/0 (socket says 172.21.15.150:60754) 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc61affd700 1 -- 172.21.15.150:0/186555766 learned_addr learned my addr 172.21.15.150:0/186555766 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc61affd700 1 -- 172.21.15.150:0/186555766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c19c270 con 0x7fc61c107cf0 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc61affd700 1 --2- 172.21.15.150:0/186555766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc60c003db0 tx=0x7fc60c003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc60c00be50 con 0x7fc61c107cf0 2024-03-18T15:50:02.549 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc60c004330 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c19c500 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc60c01a5c0 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c19f1f0 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.942+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc60c01ab50 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.943+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc60c024730 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.943+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c0693d0 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:01.945+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc60c0119c0 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.012+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc61c19f3c0 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.012+0000 7fc62096c700 1 -- 172.21.15.150:0/186555766 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7fc60c011470 con 0x7fc61c107cf0 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 msgr2=0x7fc61c19bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 --2- 172.21.15.150:0/186555766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc60c003db0 tx=0x7fc60c003e90 comp rx=0 tx=0).stop 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 shutdown_connections 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 --2- 172.21.15.150:0/186555766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc61c107cf0 0x7fc61c19bd30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:02.550 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 >> 172.21.15.150:0/186555766 conn(0x7fc61c1032b0 msgr2=0x7fc61c103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:02.551 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 shutdown_connections 2024-03-18T15:50:02.551 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:02.013+0000 7fc62196e700 1 -- 172.21.15.150:0/186555766 wait complete. 2024-03-18T15:50:02.551 INFO:teuthology.orchestra.run.smithi150.stdout:Restarting the monitor... 2024-03-18T15:50:02.808 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:02 smithi150 systemd[1]: Stopping Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:50:03.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a[31561]: 2024-03-18T15:50:02.807+0000 7fe134c08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:50:03.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a[31561]: 2024-03-18T15:50:02.807+0000 7fe134c08700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T15:50:03.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 podman[31820]: 2024-03-18 15:50:03.053217009 +0000 UTC m=+0.350164697 container died 5c737e9791b22cc6f0ba3dcacebf31ec17bd4631195a7b699dbf444d5bf8c03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T15:50:03.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 podman[31820]: 2024-03-18 15:50:03.348703148 +0000 UTC m=+0.645650847 container cleanup 5c737e9791b22cc6f0ba3dcacebf31ec17bd4631195a7b699dbf444d5bf8c03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:50:03.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 bash[31820]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a 2024-03-18T15:50:03.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 podman[31834]: 2024-03-18 15:50:03.566370112 +0000 UTC m=+0.512305924 container remove 5c737e9791b22cc6f0ba3dcacebf31ec17bd4631195a7b699dbf444d5bf8c03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T15:50:03.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T15:50:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 systemd[1]: Stopped Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:50:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 systemd[1]: Starting Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:50:04.163 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 podman[31927]: 2024-03-18 15:50:03.806812592 +0000 UTC m=+0.017694935 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:50:04.163 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:03 smithi150 podman[31927]: 2024-03-18 15:50:03.93397828 +0000 UTC m=+0.144860608 container create 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T15:50:04.163 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 podman[31927]: 2024-03-18 15:50:04.136407977 +0000 UTC m=+0.347290316 container init 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-18T15:50:04.163 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 podman[31927]: 2024-03-18 15:50:04.139234538 +0000 UTC m=+0.350116876 container start 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:50:04.257 INFO:teuthology.orchestra.run.smithi150.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T15:50:04.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:50:04.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:50:04.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: pidfile_write: ignore empty --pid-file 2024-03-18T15:50:04.472 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: load: jerasure load: lrc 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Git sha 0 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: DB SUMMARY 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: DB Session ID: 70IIVSVM2O4UWRIREC49 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105350 ; 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:50:04.473 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.env: 0x5613356fdc60 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.info_log: 0x561337601ac0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.statistics: (nil) 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.use_fsync: 0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:50:04.474 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.db_log_dir: 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.wal_dir: 2024-03-18T15:50:04.475 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:50:04.476 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.write_buffer_manager: 0x5613375d41e0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:50:04.477 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.unordered_write: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.row_cache: None 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.wal_filter: None 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:50:04.478 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.wal_compression: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:50:04.479 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_open_files: -1 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:50:04.480 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Compression algorithms supported: 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kZSTD supported: 0 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:50:04.481 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.merge_operator: 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_filter: None 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561337601bc0) 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_type: 0 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: data_block_index_type: 0 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_shortening: 1 2024-03-18T15:50:04.482 INFO:journalctl@ceph.mon.a.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: checksum: 4 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: no_block_cache: 0 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache: 0x561337607090 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_options: 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: capacity : 536870912 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: num_shard_bits : 4 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: strict_capacity_limit : 0 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_cache_compressed: (nil) 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: persistent_cache: (nil) 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_size: 4096 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_size_deviation: 10 2024-03-18T15:50:04.483 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_restart_interval: 16 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: index_block_restart_interval: 1 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: metadata_block_size: 4096 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: partition_filters: 0 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: use_delta_encoding: 1 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: filter_policy: bloomfilter 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: whole_key_filtering: 1 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: verify_compression: 0 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: format_version: 5 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: enable_index_compression: 1 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: block_align: 0 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: max_auto_readahead_size: 262144 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: prepopulate_block_cache: 0 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:50:04.484 INFO:journalctl@ceph.mon.a.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression: NoCompression 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.num_levels: 7 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:50:04.485 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:50:04.486 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:50:04.487 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:50:04.488 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:50:04.489 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.ttl: 2592000 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:50:04.490 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fb95d278-f012-4168-a31b-70bee0947838 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777004169285, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777004170302, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100461, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98372, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14219, "raw_average_key_size": 49, "raw_value_size": 90636, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710777004, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fb95d278-f012-4168-a31b-70bee0947838", "db_session_id": "70IIVSVM2O4UWRIREC49", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777004170386, "job": 1, "event": "recovery_finished"} 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5613376fa000 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: DB pointer 0x5613376e4000 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** DB Stats ** 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:50:04.491 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:50:04.492 INFO:journalctl@ceph.mon.a.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: L0 2/0 99.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Sum 2/0 99.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** Compaction Stats [default] ** 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:50:04.493 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: Cumulative compaction: 0.00 GB write, 31.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: Interval compaction: 0.00 GB write, 31.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: Block cache BinnedLRUCache@0x561337607090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: starting mon.a rank 0 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???) e1 preinit fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).mds e1 new map 2024-03-18T15:50:04.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).mds e1 print_map 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: e1 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: legacy client fscid: -1 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout: No filesystems configured 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: monmap epoch 1 2024-03-18T15:50:04.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: last_changed 2024-03-18T15:49:56.417770+0000 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: min_mon_release 19 (squid) 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: election_strategy: 1 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: fsmap 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 ceph-mon[31946]: mgrmap e1: no daemons active 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 bash[31927]: 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 2024-03-18T15:50:04.496 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:04 smithi150 systemd[1]: Started Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:50:05.330 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.674+0000 7f7807455700 1 Processor -- start 2024-03-18T15:50:05.330 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.674+0000 7f7807455700 1 -- start start 2024-03-18T15:50:05.334 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f7807455700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:05.334 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f7807455700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f780007c730 con 0x7f7800108730 2024-03-18T15:50:05.334 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f78051f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f78051f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53178/0 (socket says 172.21.15.150:53178) 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f78051f1700 1 -- 172.21.15.150:0/3200414745 learned_addr learned my addr 172.21.15.150:0/3200414745 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f78051f1700 1 -- 172.21.15.150:0/3200414745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780010b060 con 0x7f7800108730 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.675+0000 7f78051f1700 1 --2- 172.21.15.150:0/3200414745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009e80 tx=0x7f77fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e699853b3791312 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f77f7fff700 1 -- 172.21.15.150:0/3200414745 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc017070 con 0x7f7800108730 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f77f7fff700 1 -- 172.21.15.150:0/3200414745 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f77fc00b8c0 con 0x7f7800108730 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f77f7fff700 1 -- 172.21.15.150:0/3200414745 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc0045c0 con 0x7f7800108730 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 -- 172.21.15.150:0/3200414745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 msgr2=0x7f780010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 --2- 172.21.15.150:0/3200414745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009e80 tx=0x7f77fc00b330 comp rx=0 tx=0).stop 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 -- 172.21.15.150:0/3200414745 shutdown_connections 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 --2- 172.21.15.150:0/3200414745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f780010ab20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 -- 172.21.15.150:0/3200414745 >> 172.21.15.150:0/3200414745 conn(0x7f7800102120 msgr2=0x7f78001045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 -- 172.21.15.150:0/3200414745 shutdown_connections 2024-03-18T15:50:05.335 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.676+0000 7f7807455700 1 -- 172.21.15.150:0/3200414745 wait complete. 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f7807455700 1 Processor -- start 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f7807455700 1 -- start start 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f7807455700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f7807455700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f780010c090 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f78051f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f78051f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53180/0 (socket says 172.21.15.150:53180) 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f78051f1700 1 -- 172.21.15.150:0/2763047264 learned_addr learned my addr 172.21.15.150:0/2763047264 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.677+0000 7f78051f1700 1 -- 172.21.15.150:0/2763047264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78001084f0 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f78051f1700 1 --2- 172.21.15.150:0/2763047264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009530 tx=0x7f77fc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc020020 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f77fc003b50 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78001066e0 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77fc017040 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7800106c20 con 0x7f7800108730 2024-03-18T15:50:05.336 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f77fc023020 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.678+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f77fc00f050 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.679+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7800063900 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.681+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f77fc02a020 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.747+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f7800106e70 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.749+0000 7f77f67fc700 1 -- 172.21.15.150:0/2763047264 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f77fc00b860 con 0x7f7800108730 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.749+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 msgr2=0x7f7800107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.749+0000 7f7807455700 1 --2- 172.21.15.150:0/2763047264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009530 tx=0x7f77fc00bd50 comp rx=0 tx=0).stop 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.750+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 shutdown_connections 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.750+0000 7f7807455700 1 --2- 172.21.15.150:0/2763047264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800108730 0x7f7800107fb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.750+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 >> 172.21.15.150:0/2763047264 conn(0x7f7800102120 msgr2=0x7f7800102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.750+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 shutdown_connections 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:04.750+0000 7f7807455700 1 -- 172.21.15.150:0/2763047264 wait complete. 2024-03-18T15:50:05.337 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T15:50:05.351 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:50:05.351 INFO:teuthology.orchestra.run.smithi150.stdout:Creating mgr... 2024-03-18T15:50:05.351 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T15:50:05.352 INFO:teuthology.orchestra.run.smithi150.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T15:50:05.584 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x 2024-03-18T15:50:05.584 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T15:50:05.737 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715.target.wants/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@.service. 2024-03-18T15:50:05.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:05 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2763047264' entity='client.admin' 2024-03-18T15:50:05.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:05 smithi150 systemd[1]: Starting Ceph mgr.x for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:50:06.221 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:50:06.222 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-03-18T15:50:06.227 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:50:06.228 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:50:06.228 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-03-18T15:50:06.228 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:50:06.235 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:50:06.235 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr to start... 2024-03-18T15:50:06.242 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr... 2024-03-18T15:50:06.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:05 smithi150 podman[32164]: 2024-03-18 15:50:05.875793034 +0000 UTC m=+0.017011347 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:50:06.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:05 smithi150 podman[32164]: 2024-03-18 15:50:05.978970331 +0000 UTC m=+0.120188630 container create 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T15:50:06.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 podman[32164]: 2024-03-18 15:50:06.139608656 +0000 UTC m=+0.280826955 container init 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-18T15:50:06.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 podman[32164]: 2024-03-18 15:50:06.142293485 +0000 UTC m=+0.283511792 container start 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T15:50:06.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 bash[32164]: 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 2024-03-18T15:50:06.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 systemd[1]: Started Ceph mgr.x for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:50:07.046 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:06 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3674877499' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:50:07.046 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:06.703+0000 7f46acf63200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:50:07.046 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:06.799+0000 7f46acf63200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:50:07.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:07.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:07.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:50:07.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:50:07.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:07.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:50:07.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:50:07.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:50:07.371 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:49:59.321001+0000", 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:07.372 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.684+0000 7f3e4e123700 1 Processor -- start 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.684+0000 7f3e4e123700 1 -- start start 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.684+0000 7f3e4e123700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.684+0000 7f3e4e123700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48108ba0 con 0x7f3e481081b0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53206/0 (socket says 172.21.15.150:53206) 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e477fe700 1 -- 172.21.15.150:0/1569831533 learned_addr learned my addr 172.21.15.150:0/1569831533 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e477fe700 1 -- 172.21.15.150:0/1569831533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e48106dc0 con 0x7f3e481081b0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e477fe700 1 --2- 172.21.15.150:0/1569831533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098a0 tx=0x7f3e38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3131a6f2a493af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.685+0000 7f3e467fc700 1 -- 172.21.15.150:0/1569831533 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e38004160 con 0x7f3e481081b0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e467fc700 1 -- 172.21.15.150:0/1569831533 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3e38004300 con 0x7f3e481081b0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e467fc700 1 -- 172.21.15.150:0/1569831533 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e38011b50 con 0x7f3e481081b0 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 -- 172.21.15.150:0/1569831533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 msgr2=0x7f3e481085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 --2- 172.21.15.150:0/1569831533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098a0 tx=0x7f3e38009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:07.373 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 -- 172.21.15.150:0/1569831533 shutdown_connections 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 --2- 172.21.15.150:0/1569831533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e481085d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 -- 172.21.15.150:0/1569831533 >> 172.21.15.150:0/1569831533 conn(0x7f3e481018b0 msgr2=0x7f3e48103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 -- 172.21.15.150:0/1569831533 shutdown_connections 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.686+0000 7f3e4e123700 1 -- 172.21.15.150:0/1569831533 wait complete. 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e4e123700 1 Processor -- start 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e4e123700 1 -- start start 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e4e123700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e4e123700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4810b180 con 0x7f3e481081b0 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53212/0 (socket says 172.21.15.150:53212) 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.687+0000 7f3e477fe700 1 -- 172.21.15.150:0/3674877499 learned_addr learned my addr 172.21.15.150:0/3674877499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e477fe700 1 -- 172.21.15.150:0/3674877499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4819e4f0 con 0x7f3e481081b0 2024-03-18T15:50:07.374 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e477fe700 1 --2- 172.21.15.150:0/3674877499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009fb0 tx=0x7f3e38011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e380038b0 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3e38011e50 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e38019550 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4819e780 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e481a1470 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3e38019b90 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.688+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3e38024730 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.689+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e480693d0 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.693+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3e38003a50 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.762+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3e480008d0 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.762+0000 7f3e44ff9700 1 -- 172.21.15.150:0/3674877499 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3e380249d0 con 0x7f3e481081b0 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.763+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 msgr2=0x7f3e4819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:07.375 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.763+0000 7f3e4e123700 1 --2- 172.21.15.150:0/3674877499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009fb0 tx=0x7f3e38011bf0 comp rx=0 tx=0).stop 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.763+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 shutdown_connections 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.763+0000 7f3e4e123700 1 --2- 172.21.15.150:0/3674877499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e481081b0 0x7f3e4819dfb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.763+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 >> 172.21.15.150:0/3674877499 conn(0x7f3e481018b0 msgr2=0x7f3e48102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.764+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 shutdown_connections 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:06.764+0000 7f3e4e123700 1 -- 172.21.15.150:0/3674877499 wait complete. 2024-03-18T15:50:07.376 INFO:teuthology.orchestra.run.smithi150.stdout:mgr not available, waiting (1/15)... 2024-03-18T15:50:07.678 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:07.447+0000 7f46acf63200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:50:07.678 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:07.538+0000 7f46acf63200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:50:07.678 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:07.677+0000 7f46acf63200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:50:08.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:07.771+0000 7f46acf63200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:50:08.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:07.851+0000 7f46acf63200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:50:08.392 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:08.110+0000 7f46acf63200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:50:08.392 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:08.216+0000 7f46acf63200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:08.392 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:08.299+0000 7f46acf63200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:50:08.392 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:08.391+0000 7f46acf63200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:50:08.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:08.565+0000 7f46acf63200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:50:09.819 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:09.525+0000 7f46acf63200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:50:09.819 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:09.657+0000 7f46acf63200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:50:10.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:09 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1175299823' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:50:10.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:10.001+0000 7f46acf63200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:50:10.450 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:10.090+0000 7f46acf63200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:50:10.450 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:10.286+0000 7f46acf63200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:10.768 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:10.480+0000 7f46acf63200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:50:11.035 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:10.767+0000 7f46acf63200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:11.036 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:11.035+0000 7f46acf63200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:11.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:11.357+0000 7f46acf63200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:50:12.591 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:12.164+0000 7f46acf63200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:50:13.060 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:12.742+0000 7f46acf63200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:50:13.060 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:12.841+0000 7f46acf63200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:50:13.060 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:13.059+0000 7f46acf63200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:50:13.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: Activating manager daemon x 2024-03-18T15:50:13.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: mgrmap e2: x(active, starting, since 0.00303571s) 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: Manager daemon x is now available 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' 2024-03-18T15:50:13.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:13 smithi150 ceph-mon[31946]: from='mgr.14100 172.21.15.150:0/1495582836' entity='mgr.x' 2024-03-18T15:50:15.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:15 smithi150 ceph-mon[31946]: mgrmap e3: x(active, since 1.00755s) 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:50:16.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:16.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:50:16.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:50:16.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:50:16.231 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:16.232 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:49:59.321001+0000", 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.818+0000 7fa8d9362700 1 Processor -- start 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.818+0000 7fa8d9362700 1 -- start start 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.818+0000 7fa8d9362700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.818+0000 7fa8d9362700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d407d150 con 0x7fa8d407e720 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.818+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:16.233 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53222/0 (socket says 172.21.15.150:53222) 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d2ffd700 1 -- 172.21.15.150:0/3954634736 learned_addr learned my addr 172.21.15.150:0/3954634736 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d2ffd700 1 -- 172.21.15.150:0/3954634736 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d407d2d0 con 0x7fa8d407e720 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d2ffd700 1 --2- 172.21.15.150:0/3954634736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e80 tx=0x7fa8cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db40498dc63c5e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d1ffb700 1 -- 172.21.15.150:0/3954634736 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8cc017070 con 0x7fa8d407e720 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d1ffb700 1 -- 172.21.15.150:0/3954634736 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa8cc00b8c0 con 0x7fa8d407e720 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.819+0000 7fa8d1ffb700 1 -- 172.21.15.150:0/3954634736 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8cc0045c0 con 0x7fa8d407e720 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 -- 172.21.15.150:0/3954634736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 msgr2=0x7fa8d407cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 --2- 172.21.15.150:0/3954634736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e80 tx=0x7fa8cc00b330 comp rx=0 tx=0).stop 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 -- 172.21.15.150:0/3954634736 shutdown_connections 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 --2- 172.21.15.150:0/3954634736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d407cb80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 -- 172.21.15.150:0/3954634736 >> 172.21.15.150:0/3954634736 conn(0x7fa8d4103230 msgr2=0x7fa8d4105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 -- 172.21.15.150:0/3954634736 shutdown_connections 2024-03-18T15:50:16.234 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.820+0000 7fa8d9362700 1 -- 172.21.15.150:0/3954634736 wait complete. 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d9362700 1 Processor -- start 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d9362700 1 -- start start 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d9362700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d9362700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d410e2c0 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53236/0 (socket says 172.21.15.150:53236) 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d2ffd700 1 -- 172.21.15.150:0/1175299823 learned_addr learned my addr 172.21.15.150:0/1175299823 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.821+0000 7fa8d2ffd700 1 -- 172.21.15.150:0/1175299823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d419e410 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8d2ffd700 1 --2- 172.21.15.150:0/1175299823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e50 tx=0x7fa8cc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8cc020020 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d419e6a0 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa8cc003b50 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8cc017040 con 0x7fa8d407e720 2024-03-18T15:50:16.235 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d41a1390 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa8cc023020 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.822+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa8cc00f050 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.823+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d40693d0 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.825+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa8cc02a020 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.897+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa8d41a1560 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.897+0000 7fa8cbfff700 1 -- 172.21.15.150:0/1175299823 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa8d41a1560 con 0x7fa8d407e720 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 msgr2=0x7fa8d419ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 --2- 172.21.15.150:0/1175299823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e50 tx=0x7fa8cc00bd30 comp rx=0 tx=0).stop 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 shutdown_connections 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 --2- 172.21.15.150:0/1175299823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8d407e720 0x7fa8d419ded0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 >> 172.21.15.150:0/1175299823 conn(0x7fa8d4103230 msgr2=0x7fa8d4103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 shutdown_connections 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:09.898+0000 7fa8d9362700 1 -- 172.21.15.150:0/1175299823 wait complete. 2024-03-18T15:50:16.236 INFO:teuthology.orchestra.run.smithi150.stdout:mgr not available, waiting (2/15)... 2024-03-18T15:50:17.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:17 smithi150 ceph-mon[31946]: mgrmap e4: x(active, since 3s) 2024-03-18T15:50:20.919 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:20 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2362571406' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:50:21.217 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:50:21.218 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:50:21.219 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:50:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:50:21.221 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:49:59.321001+0000", 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.338+0000 7f872b01e700 1 Processor -- start 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.338+0000 7f872b01e700 1 -- start start 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.338+0000 7f872b01e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.338+0000 7f872b01e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87241086e0 con 0x7f8724107cf0 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f8728dba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f8728dba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53312/0 (socket says 172.21.15.150:53312) 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f8728dba700 1 -- 172.21.15.150:0/4244487483 learned_addr learned my addr 172.21.15.150:0/4244487483 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f8728dba700 1 -- 172.21.15.150:0/4244487483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8724108ef0 con 0x7f8724107cf0 2024-03-18T15:50:21.222 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f8728dba700 1 --2- 172.21.15.150:0/4244487483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b180 tx=0x7f871c00b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0ad66312d80493 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f87237fe700 1 -- 172.21.15.150:0/4244487483 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f871c017070 con 0x7f8724107cf0 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f87237fe700 1 -- 172.21.15.150:0/4244487483 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f871c003a20 con 0x7f8724107cf0 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.339+0000 7f87237fe700 1 -- 172.21.15.150:0/4244487483 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f871c004820 con 0x7f8724107cf0 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 -- 172.21.15.150:0/4244487483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 msgr2=0x7f8724108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 --2- 172.21.15.150:0/4244487483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b180 tx=0x7f871c00b490 comp rx=0 tx=0).stop 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 -- 172.21.15.150:0/4244487483 shutdown_connections 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 --2- 172.21.15.150:0/4244487483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f8724108110 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 -- 172.21.15.150:0/4244487483 >> 172.21.15.150:0/4244487483 conn(0x7f87241032b0 msgr2=0x7f87241056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 -- 172.21.15.150:0/4244487483 shutdown_connections 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.340+0000 7f872b01e700 1 -- 172.21.15.150:0/4244487483 wait complete. 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f872b01e700 1 Processor -- start 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f872b01e700 1 -- start start 2024-03-18T15:50:21.223 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f872b01e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f872b01e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f872410c2a0 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8728dba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8728dba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53314/0 (socket says 172.21.15.150:53314) 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8728dba700 1 -- 172.21.15.150:0/2362571406 learned_addr learned my addr 172.21.15.150:0/2362571406 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8728dba700 1 -- 172.21.15.150:0/2362571406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872407e930 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8728dba700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f871c0045b0 tx=0x7f871c003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f871c020020 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f871c00be50 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872407cb20 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.341+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f871c017040 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.342+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872407d060 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f871c023020 con 0x7f8724107cf0 2024-03-18T15:50:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f8721ffb700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 0x7f871003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f871c00f050 con 0x7f8724107cf0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87240693d0 con 0x7f8724107cf0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f8723fff700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 0x7f871003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.343+0000 7f8723fff700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 0x7f871003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8714006fd0 tx=0x7f8714006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.348+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f871c02a020 con 0x7f8724107cf0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.620+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f872407d280 con 0x7f8724107cf0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.620+0000 7f8721ffb700 1 -- 172.21.15.150:0/2362571406 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f871c02c300 con 0x7f8724107cf0 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 msgr2=0x7f871003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 0x7f871003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8714006fd0 tx=0x7f8714006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 msgr2=0x7f872407e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:21.225 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f871c0045b0 tx=0x7f871c003eb0 comp rx=0 tx=0).stop 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 shutdown_connections 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f8710038e80 0x7f871003b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 --2- 172.21.15.150:0/2362571406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8724107cf0 0x7f872407e3f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 >> 172.21.15.150:0/2362571406 conn(0x7f87241032b0 msgr2=0x7f8724103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 shutdown_connections 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:20.622+0000 7f872b01e700 1 -- 172.21.15.150:0/2362571406 wait complete. 2024-03-18T15:50:21.226 INFO:teuthology.orchestra.run.smithi150.stdout:mgr is available 2024-03-18T15:50:22.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:21 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1410959600' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout fsid = 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:50:22.360 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e330ee700 1 Processor -- start 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e330ee700 1 -- start start 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e330ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e330ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c07be60 con 0x7f0e2c108780 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e30e8a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e30e8a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53320/0 (socket says 172.21.15.150:53320) 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e30e8a700 1 -- 172.21.15.150:0/2193374085 learned_addr learned my addr 172.21.15.150:0/2193374085 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:22.361 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e30e8a700 1 -- 172.21.15.150:0/2193374085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c10b0b0 con 0x7f0e2c108780 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.626+0000 7f0e30e8a700 1 --2- 172.21.15.150:0/2193374085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e200098a0 tx=0x7f0e20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e2e2a76b93f558 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e2b7fe700 1 -- 172.21.15.150:0/2193374085 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e20004160 con 0x7f0e2c108780 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e2b7fe700 1 -- 172.21.15.150:0/2193374085 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e20004300 con 0x7f0e2c108780 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e2b7fe700 1 -- 172.21.15.150:0/2193374085 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e20011b50 con 0x7f0e2c108780 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 -- 172.21.15.150:0/2193374085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 msgr2=0x7f0e2c10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 --2- 172.21.15.150:0/2193374085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e200098a0 tx=0x7f0e20009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 -- 172.21.15.150:0/2193374085 shutdown_connections 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 --2- 172.21.15.150:0/2193374085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c10ab70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 -- 172.21.15.150:0/2193374085 >> 172.21.15.150:0/2193374085 conn(0x7f0e2c102130 msgr2=0x7f0e2c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 -- 172.21.15.150:0/2193374085 shutdown_connections 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.627+0000 7f0e330ee700 1 -- 172.21.15.150:0/2193374085 wait complete. 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e330ee700 1 Processor -- start 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e330ee700 1 -- start start 2024-03-18T15:50:22.362 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e330ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e330ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10c0d0 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e30e8a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e30e8a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53330/0 (socket says 172.21.15.150:53330) 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e30e8a700 1 -- 172.21.15.150:0/1410959600 learned_addr learned my addr 172.21.15.150:0/1410959600 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.628+0000 7f0e30e8a700 1 -- 172.21.15.150:0/1410959600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c19c210 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e30e8a700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009fb0 tx=0x7f0e20011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e200038b0 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c19c4a0 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e20011e50 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e20019420 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.629+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c19f180 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f0e20019720 con 0x7f0e2c108780 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e29ffb700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 0x7f0e1c03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:22.363 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c0693d0 con 0x7f0e2c108780 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0e2004cfc0 con 0x7f0e2c108780 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e2bfff700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 0x7f0e1c03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.630+0000 7f0e2bfff700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 0x7f0e1c03b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0e18006fd0 tx=0x7f0e18006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.635+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e20003a50 con 0x7f0e2c108780 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.825+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0e2c19f350 con 0x7f0e2c108780 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.826+0000 7f0e29ffb700 1 -- 172.21.15.150:0/1410959600 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f0e2c19f350 con 0x7f0e2c108780 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 msgr2=0x7f0e1c03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 0x7f0e1c03b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0e18006fd0 tx=0x7f0e18006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 msgr2=0x7f0e2c19bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:22.364 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009fb0 tx=0x7f0e20011bf0 comp rx=0 tx=0).stop 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 shutdown_connections 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f0e1c038e80 0x7f0e1c03b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 --2- 172.21.15.150:0/1410959600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e2c108780 0x7f0e2c19bcd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 >> 172.21.15.150:0/1410959600 conn(0x7f0e2c102130 msgr2=0x7f0e2c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 shutdown_connections 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:21.829+0000 7f0e330ee700 1 -- 172.21.15.150:0/1410959600 wait complete. 2024-03-18T15:50:22.365 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling cephadm module... 2024-03-18T15:50:23.324 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:23 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3182340387' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T15:50:23.325 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:23 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setuser ceph since I am not root 2024-03-18T15:50:23.325 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:23 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setgroup ceph since I am not root 2024-03-18T15:50:23.607 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.710+0000 7fbd7ad87700 1 Processor -- start 2024-03-18T15:50:23.607 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd7ad87700 1 -- start start 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd7ad87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd7ad87700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7407c730 con 0x7fbd74108730 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd78b23700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd78b23700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37812/0 (socket says 172.21.15.150:37812) 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd78b23700 1 -- 172.21.15.150:0/3965058966 learned_addr learned my addr 172.21.15.150:0/3965058966 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd78b23700 1 -- 172.21.15.150:0/3965058966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7410b060 con 0x7fbd74108730 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.711+0000 7fbd78b23700 1 --2- 172.21.15.150:0/3965058966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd680098a0 tx=0x7fbd68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669fc5c56c64ed1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd737fe700 1 -- 172.21.15.150:0/3965058966 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd68004160 con 0x7fbd74108730 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd737fe700 1 -- 172.21.15.150:0/3965058966 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbd68004300 con 0x7fbd74108730 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd737fe700 1 -- 172.21.15.150:0/3965058966 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd68011b50 con 0x7fbd74108730 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3965058966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 msgr2=0x7fbd7410ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3965058966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd680098a0 tx=0x7fbd68009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3965058966 shutdown_connections 2024-03-18T15:50:23.608 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3965058966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd7410ab20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3965058966 >> 172.21.15.150:0/3965058966 conn(0x7fbd74102160 msgr2=0x7fbd741045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3965058966 shutdown_connections 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.712+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3965058966 wait complete. 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd7ad87700 1 Processor -- start 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd7ad87700 1 -- start start 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd7ad87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd7ad87700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7410c090 con 0x7fbd74108730 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd78b23700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd78b23700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37814/0 (socket says 172.21.15.150:37814) 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd78b23700 1 -- 172.21.15.150:0/3182340387 learned_addr learned my addr 172.21.15.150:0/3182340387 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.713+0000 7fbd78b23700 1 -- 172.21.15.150:0/3182340387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd741084f0 con 0x7fbd74108730 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd78b23700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbd68011b60 tx=0x7fbd68011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd68003970 con 0x7fbd74108730 2024-03-18T15:50:23.609 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd741066e0 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbd68003b10 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd68024610 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.714+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd74106c20 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.715+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fbd68024910 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.715+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd74063900 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.715+0000 7fbd71ffb700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 0x7fbd6403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.715+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbd68011e00 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.715+0000 7fbd73fff700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 0x7fbd6403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.716+0000 7fbd73fff700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 0x7fbd6403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbd60006fd0 tx=0x7fbd60006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.720+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd6800ba00 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:22.969+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fbd74106ea0 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.020+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fbd6800bc60 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.022+0000 7fbd71ffb700 1 -- 172.21.15.150:0/3182340387 <== mon.0 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fbd68052020 con 0x7fbd74108730 2024-03-18T15:50:23.610 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 msgr2=0x7fbd6403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 0x7fbd6403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbd60006fd0 tx=0x7fbd60006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 msgr2=0x7fbd74107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbd68011b60 tx=0x7fbd68011c40 comp rx=0 tx=0).stop 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 shutdown_connections 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fbd64038ed0 0x7fbd6403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 --2- 172.21.15.150:0/3182340387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd74108730 0x7fbd74107fb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 >> 172.21.15.150:0/3182340387 conn(0x7fbd74102160 msgr2=0x7fbd74102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 shutdown_connections 2024-03-18T15:50:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:23.024+0000 7fbd7ad87700 1 -- 172.21.15.150:0/3182340387 wait complete. 2024-03-18T15:50:23.839 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:23 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:23.582+0000 7f5405671200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:50:23.839 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:23 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:23.686+0000 7f5405671200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:50:24.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3182340387' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T15:50:24.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-mon[31946]: mgrmap e5: x(active, since 9s) 2024-03-18T15:50:24.531 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:24.357+0000 7f5405671200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:50:24.531 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:24.451+0000 7f5405671200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:50:24.806 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:24.597+0000 7f5405671200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:50:24.806 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:24.694+0000 7f5405671200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:50:24.807 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:24.779+0000 7f5405671200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:50:24.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:24.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T15:50:24.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e855f2700 1 Processor -- start 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e855f2700 1 -- start start 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e855f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e855f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8007c670 con 0x7f5e801087e0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37852/0 (socket says 172.21.15.150:37852) 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e7effd700 1 -- 172.21.15.150:0/63771222 learned_addr learned my addr 172.21.15.150:0/63771222 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.034+0000 7f5e7effd700 1 -- 172.21.15.150:0/63771222 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8010b110 con 0x7f5e801087e0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e7effd700 1 --2- 172.21.15.150:0/63771222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e700098a0 tx=0x7f5e70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d94b34064b79b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e7dffb700 1 -- 172.21.15.150:0/63771222 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e70004160 con 0x7f5e801087e0 2024-03-18T15:50:24.815 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e7dffb700 1 -- 172.21.15.150:0/63771222 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5e70004300 con 0x7f5e801087e0 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e7dffb700 1 -- 172.21.15.150:0/63771222 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e70011b50 con 0x7f5e801087e0 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e855f2700 1 -- 172.21.15.150:0/63771222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 msgr2=0x7f5e8010abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.035+0000 7f5e855f2700 1 --2- 172.21.15.150:0/63771222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e700098a0 tx=0x7f5e70009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 -- 172.21.15.150:0/63771222 shutdown_connections 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 --2- 172.21.15.150:0/63771222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8010abd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 -- 172.21.15.150:0/63771222 >> 172.21.15.150:0/63771222 conn(0x7f5e80102190 msgr2=0x7f5e801045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 -- 172.21.15.150:0/63771222 shutdown_connections 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 -- 172.21.15.150:0/63771222 wait complete. 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 Processor -- start 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.036+0000 7f5e855f2700 1 -- start start 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e855f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e855f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8010c080 con 0x7f5e801087e0 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:24.816 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37854/0 (socket says 172.21.15.150:37854) 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e7effd700 1 -- 172.21.15.150:0/2127030317 learned_addr learned my addr 172.21.15.150:0/2127030317 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e7effd700 1 -- 172.21.15.150:0/2127030317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8019c210 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e7effd700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009fb0 tx=0x7f5e70011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e700038b0 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5e70011e50 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8019c4a0 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e70019420 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.037+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8019f180 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.038+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e800693d0 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.038+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f5e70019720 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.038+0000 7f5e67fff700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 0x7f5e6803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.039+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5e7004d060 con 0x7f5e801087e0 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.039+0000 7f5e7e7fc700 1 -- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 msgr2=0x7f5e6803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:24.817 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.039+0000 7f5e7e7fc700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 0x7f5e6803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.043+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e7000bc70 con 0x7f5e801087e0 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.240+0000 7f5e7e7fc700 1 -- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 msgr2=0x7f5e6803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.240+0000 7f5e7e7fc700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 0x7f5e6803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.303+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5e8019f350 con 0x7f5e801087e0 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.303+0000 7f5e67fff700 1 -- 172.21.15.150:0/2127030317 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f5e8019f350 con 0x7f5e801087e0 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 msgr2=0x7f5e6803b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 0x7f5e6803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 msgr2=0x7f5e8019bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009fb0 tx=0x7f5e70011bf0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 shutdown_connections 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7f5e68038ed0 0x7f5e6803b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 --2- 172.21.15.150:0/2127030317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e801087e0 0x7f5e8019bcd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 >> 172.21.15.150:0/2127030317 conn(0x7f5e80102190 msgr2=0x7f5e80102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:24.818 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.305+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 shutdown_connections 2024-03-18T15:50:24.819 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:24.306+0000 7f5e855f2700 1 -- 172.21.15.150:0/2127030317 wait complete. 2024-03-18T15:50:24.819 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for the mgr to restart... 2024-03-18T15:50:24.819 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr epoch 5... 2024-03-18T15:50:25.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2127030317' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:50:25.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:25.053+0000 7f5405671200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:50:25.521 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:25.163+0000 7f5405671200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:25.521 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:25.249+0000 7f5405671200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:50:25.522 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:25.345+0000 7f5405671200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:50:25.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:25.521+0000 7f5405671200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:50:26.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:26 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:26.483+0000 7f5405671200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:50:26.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:26 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:26.612+0000 7f5405671200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:50:27.237 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:26 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:26.947+0000 7f5405671200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:50:27.237 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:27.036+0000 7f5405671200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:50:27.237 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:27.235+0000 7f5405671200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:27.591 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:27.421+0000 7f5405671200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:50:27.967 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:27.706+0000 7f5405671200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:28.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:27.965+0000 7f5405671200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:28.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:28 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:28.198+0000 7f5405671200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:50:29.242 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:28 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:28.996+0000 7f5405671200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:50:29.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:29.567+0000 7f5405671200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:50:29.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:29.666+0000 7f5405671200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:50:30.261 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:29.881+0000 7f5405671200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: Active manager daemon x restarted 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: Activating manager daemon x 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: mgrmap e6: x(active, starting, since 0.00364032s) 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:50:30.262 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: Manager daemon x is now available 2024-03-18T15:50:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:30.263 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:29 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:31.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: Found migration_current of "None". Setting to last migration. 2024-03-18T15:50:31.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:31.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:50:31.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:50:31.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:31.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:31.222 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:30 smithi150 ceph-mon[31946]: mgrmap e7: x(active, since 1.00608s) 2024-03-18T15:50:31.498 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd892745700 1 Processor -- start 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd892745700 1 -- start start 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd892745700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd892745700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1075b0 con 0x7fd88c106bc0 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37860/0 (socket says 172.21.15.150:37860) 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd88bfff700 1 -- 172.21.15.150:0/1065329079 learned_addr learned my addr 172.21.15.150:0/1065329079 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:31.499 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.974+0000 7fd88bfff700 1 -- 172.21.15.150:0/1065329079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c107dc0 con 0x7fd88c106bc0 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd88bfff700 1 --2- 172.21.15.150:0/1065329079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00bb40 tx=0x7fd87c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454650bf01a9f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd88affd700 1 -- 172.21.15.150:0/1065329079 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd87c00dbe0 con 0x7fd88c106bc0 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd88affd700 1 -- 172.21.15.150:0/1065329079 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd87c0044f0 con 0x7fd88c106bc0 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd88affd700 1 -- 172.21.15.150:0/1065329079 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd87c010470 con 0x7fd88c106bc0 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd892745700 1 -- 172.21.15.150:0/1065329079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 msgr2=0x7fd88c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.975+0000 7fd892745700 1 --2- 172.21.15.150:0/1065329079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00bb40 tx=0x7fd87c00be50 comp rx=0 tx=0).stop 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 -- 172.21.15.150:0/1065329079 shutdown_connections 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 --2- 172.21.15.150:0/1065329079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c106fe0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 -- 172.21.15.150:0/1065329079 >> 172.21.15.150:0/1065329079 conn(0x7fd88c102120 msgr2=0x7fd88c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 -- 172.21.15.150:0/1065329079 shutdown_connections 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 -- 172.21.15.150:0/1065329079 wait complete. 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 Processor -- start 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.976+0000 7fd892745700 1 -- start start 2024-03-18T15:50:31.500 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd892745700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd892745700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c07b850 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd88bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37864/0 (socket says 172.21.15.150:37864) 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd88bfff700 1 -- 172.21.15.150:0/777758492 learned_addr learned my addr 172.21.15.150:0/777758492 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd88bfff700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c19a090 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd88bfff700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00b340 tx=0x7fd87c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd87c027070 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c19a320 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd87c0104e0 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c19cfc0 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.977+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd87c022c40 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fd87c02a070 con 0x7fd88c106bc0 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd8897fa700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:31.501 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd87403baa0 con 0x7fd874038ed0 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd87c022e00 con 0x7fd88c106bc0 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd88b7fe700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:25.978+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:26.179+0000 7fd88b7fe700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:26.179+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:26.580+0000 7fd88b7fe700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:26.580+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:27.381+0000 7fd88b7fe700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:27.381+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:28.982+0000 7fd88b7fe700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/2001484430 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:28.982+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:29.896+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fd87c02cce0 con 0x7fd88c106bc0 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:29.896+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 msgr2=0x7fd87403b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:50:31.502 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:29.896+0000 7fd8897fa700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.898+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fd87c050e50 con 0x7fd88c106bc0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.898+0000 7fd8897fa700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 0x7fd87403ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.898+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd87403baa0 con 0x7fd87403c7c0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.898+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 0x7fd87403ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.899+0000 7fd88b7fe700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 0x7fd87403ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd880003a10 tx=0x7fd880009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.899+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd87403baa0 con 0x7fd87403c7c0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd88c0693d0 con 0x7fd87403c7c0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd8897fa700 1 -- 172.21.15.150:0/777758492 <== mgr.14118 v2:172.21.15.150:6800/775411148 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd88c0693d0 con 0x7fd87403c7c0 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 msgr2=0x7fd87403ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 0x7fd87403ebb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd880003a10 tx=0x7fd880009290 comp rx=0 tx=0).stop 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 msgr2=0x7fd88c199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00b340 tx=0x7fd87c0040b0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 shutdown_connections 2024-03-18T15:50:31.503 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fd87403c7c0 0x7fd87403ebb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88c106bc0 0x7fd88c199b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 --2- 172.21.15.150:0/777758492 >> [v2:172.21.15.150:6800/2001484430,v1:172.21.15.150:6801/2001484430] conn(0x7fd874038ed0 0x7fd87403b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 >> 172.21.15.150:0/777758492 conn(0x7fd88c102120 msgr2=0x7fd88c190160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 shutdown_connections 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:30.906+0000 7fd892745700 1 -- 172.21.15.150:0/777758492 wait complete. 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:mgr epoch 5 is available 2024-03-18T15:50:31.504 INFO:teuthology.orchestra.run.smithi150.stdout:Setting orchestrator backend to cephadm... 2024-03-18T15:50:32.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:30] ENGINE Bus STARTING 2024-03-18T15:50:32.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:50:32.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:50:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:30] ENGINE Serving on https://172.21.15.150:7150 2024-03-18T15:50:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:31] ENGINE Serving on http://172.21.15.150:8765 2024-03-18T15:50:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:31] ENGINE Bus STARTED 2024-03-18T15:50:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:31 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:32.627 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb3329d9700 1 Processor -- start 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb3329d9700 1 -- start start 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb3329d9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb3329d9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c07d130 con 0x7fb32c07e700 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37930/0 (socket says 172.21.15.150:37930) 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb32bfff700 1 -- 172.21.15.150:0/2616949357 learned_addr learned my addr 172.21.15.150:0/2616949357 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.891+0000 7fb32bfff700 1 -- 172.21.15.150:0/2616949357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c07d2b0 con 0x7fb32c07e700 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb32bfff700 1 --2- 172.21.15.150:0/2616949357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0098a0 tx=0x7fb31c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c9f83b5a69a4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb32affd700 1 -- 172.21.15.150:0/2616949357 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb31c004160 con 0x7fb32c07e700 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb32affd700 1 -- 172.21.15.150:0/2616949357 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb31c004300 con 0x7fb32c07e700 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb32affd700 1 -- 172.21.15.150:0/2616949357 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb31c011b50 con 0x7fb32c07e700 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 -- 172.21.15.150:0/2616949357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 msgr2=0x7fb32c07cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:32.628 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 --2- 172.21.15.150:0/2616949357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0098a0 tx=0x7fb31c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 -- 172.21.15.150:0/2616949357 shutdown_connections 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 --2- 172.21.15.150:0/2616949357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c07cb60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 -- 172.21.15.150:0/2616949357 >> 172.21.15.150:0/2616949357 conn(0x7fb32c1032b0 msgr2=0x7fb32c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 -- 172.21.15.150:0/2616949357 shutdown_connections 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.892+0000 7fb3329d9700 1 -- 172.21.15.150:0/2616949357 wait complete. 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.893+0000 7fb3329d9700 1 Processor -- start 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.893+0000 7fb3329d9700 1 -- start start 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.893+0000 7fb3329d9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.893+0000 7fb3329d9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c10e300 con 0x7fb32c07e700 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.893+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37940/0 (socket says 172.21.15.150:37940) 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb32bfff700 1 -- 172.21.15.150:0/3255591610 learned_addr learned my addr 172.21.15.150:0/3255591610 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb32bfff700 1 -- 172.21.15.150:0/3255591610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c19e400 con 0x7fb32c07e700 2024-03-18T15:50:32.629 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb32bfff700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb31c011b60 tx=0x7fb31c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb31c003970 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32c19e690 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb31c003b10 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb31c024610 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.894+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32c1a1380 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.895+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fb31c0248c0 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.895+0000 7fb3297fa700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 0x7fb31403b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.895+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb31c011e00 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.895+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c063900 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.895+0000 7fb32b7fe700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 0x7fb31403b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.896+0000 7fb32b7fe700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 0x7fb31403b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb320006fd0 tx=0x7fb320006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.901+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb31c02d910 con 0x7fb32c07e700 2024-03-18T15:50:32.630 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:31.949+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fb31c02d3f0 con 0x7fb32c07e700 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.113+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb32c104530 con 0x7fb314038e50 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.120+0000 7fb3297fa700 1 -- 172.21.15.150:0/3255591610 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb32c104530 con 0x7fb314038e50 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 msgr2=0x7fb31403b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 0x7fb31403b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb320006fd0 tx=0x7fb320006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 msgr2=0x7fb32c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb31c011b60 tx=0x7fb31c011c40 comp rx=0 tx=0).stop 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 shutdown_connections 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fb314038e50 0x7fb31403b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 --2- 172.21.15.150:0/3255591610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb32c07e700 0x7fb32c19dec0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 >> 172.21.15.150:0/3255591610 conn(0x7fb32c1032b0 msgr2=0x7fb32c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.122+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 shutdown_connections 2024-03-18T15:50:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:32.123+0000 7fb3329d9700 1 -- 172.21.15.150:0/3255591610 wait complete. 2024-03-18T15:50:33.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:32 smithi150 ceph-mon[31946]: mgrmap e8: x(active, since 2s) 2024-03-18T15:50:33.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:32 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:33.039 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:32 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdf175700 1 Processor -- start 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdf175700 1 -- start start 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdf175700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdf175700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd807c6f0 con 0x7f5fd8108770 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdcf11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdcf11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55242/0 (socket says 172.21.15.150:55242) 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.037+0000 7f5fdcf11700 1 -- 172.21.15.150:0/3220048373 learned_addr learned my addr 172.21.15.150:0/3220048373 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.038+0000 7f5fdcf11700 1 -- 172.21.15.150:0/3220048373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd810b0a0 con 0x7f5fd8108770 2024-03-18T15:50:33.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.038+0000 7f5fdcf11700 1 --2- 172.21.15.150:0/3220048373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0098a0 tx=0x7f5fcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a167fad8dd91a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.038+0000 7f5fd77fe700 1 -- 172.21.15.150:0/3220048373 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc004160 con 0x7f5fd8108770 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.038+0000 7f5fd77fe700 1 -- 172.21.15.150:0/3220048373 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fcc004300 con 0x7f5fd8108770 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.038+0000 7f5fd77fe700 1 -- 172.21.15.150:0/3220048373 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc011b50 con 0x7f5fd8108770 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- 172.21.15.150:0/3220048373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 msgr2=0x7f5fd810ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 --2- 172.21.15.150:0/3220048373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0098a0 tx=0x7f5fcc009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- 172.21.15.150:0/3220048373 shutdown_connections 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 --2- 172.21.15.150:0/3220048373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd810ab60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- 172.21.15.150:0/3220048373 >> 172.21.15.150:0/3220048373 conn(0x7f5fd8102120 msgr2=0x7f5fd81045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- 172.21.15.150:0/3220048373 shutdown_connections 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- 172.21.15.150:0/3220048373 wait complete. 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 Processor -- start 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- start start 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.039+0000 7f5fdf175700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd810c0c0 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdcf11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdcf11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55250/0 (socket says 172.21.15.150:55250) 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdcf11700 1 -- 172.21.15.150:0/64806120 learned_addr learned my addr 172.21.15.150:0/64806120 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdcf11700 1 -- 172.21.15.150:0/64806120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd819c1b0 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdcf11700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc011b60 tx=0x7f5fcc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc003970 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fcc003b10 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc024610 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd819c440 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.040+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd819f130 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.041+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5fcc01a9f0 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.041+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd80693d0 con 0x7f5fd8108770 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.041+0000 7f5fd5ffb700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 0x7f5fc803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:33.812 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.041+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5fcc011e00 con 0x7f5fd8108770 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.042+0000 7f5fd7fff700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 0x7f5fc803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.042+0000 7f5fd7fff700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 0x7f5fc803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4006fd0 tx=0x7f5fc4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.047+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fcc019e10 con 0x7f5fd8108770 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.244+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5fd81033f0 con 0x7f5fc8038f70 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.244+0000 7f5fd5ffb700 1 -- 172.21.15.150:0/64806120 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5fd81033f0 con 0x7f5fc8038f70 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 msgr2=0x7f5fc803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 0x7f5fc803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4006fd0 tx=0x7f5fc4006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 msgr2=0x7f5fd819bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc011b60 tx=0x7f5fcc011c40 comp rx=0 tx=0).stop 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 shutdown_connections 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f5fc8038f70 0x7f5fc803b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 --2- 172.21.15.150:0/64806120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fd8108770 0x7f5fd819bc70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 >> 172.21.15.150:0/64806120 conn(0x7f5fd8102120 msgr2=0x7f5fd8102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:33.813 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 shutdown_connections 2024-03-18T15:50:33.814 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:33.247+0000 7f5fdf175700 1 -- 172.21.15.150:0/64806120 wait complete. 2024-03-18T15:50:33.814 INFO:teuthology.orchestra.run.smithi150.stdout:Generating ssh key... 2024-03-18T15:50:34.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:33 smithi150 ceph-mon[31946]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:34.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: Generating public/private rsa key pair. 2024-03-18T15:50:34.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: Your identification has been saved in /tmp/tmpl18tpv9j/key. 2024-03-18T15:50:34.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: Your public key has been saved in /tmp/tmpl18tpv9j/key.pub. 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: The key fingerprint is: 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: SHA256:zBP5XAN2czRasARJOzL5XKc99dHUos/y1vvvVoGYVVE ceph-0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: The key's randomart image is: 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: +---[RSA 3072]----+ 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | .=o=o*oE| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | +.= B.oo| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | * o X.ooo| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | o O *.* oo| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | S = .oo o| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | . . o..| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | o ..| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | o o| 2024-03-18T15:50:34.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: | . +B| 2024-03-18T15:50:34.843 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: +----[SHA256]-----+ 2024-03-18T15:50:35.167 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-mon[31946]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:35.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:35.168 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:34 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:35.177 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c8a899700 1 Processor -- start 2024-03-18T15:50:35.177 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c8a899700 1 -- start start 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c8a899700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c8a899700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c841075d0 con 0x7f9c84106be0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55266/0 (socket says 172.21.15.150:55266) 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.196+0000 7f9c83fff700 1 -- 172.21.15.150:0/3239921841 learned_addr learned my addr 172.21.15.150:0/3239921841 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.197+0000 7f9c83fff700 1 -- 172.21.15.150:0/3239921841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c84107de0 con 0x7f9c84106be0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.197+0000 7f9c83fff700 1 --2- 172.21.15.150:0/3239921841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009ab0 tx=0x7f9c74009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6456436b28efa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.197+0000 7f9c82ffd700 1 -- 172.21.15.150:0/3239921841 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74004160 con 0x7f9c84106be0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.197+0000 7f9c82ffd700 1 -- 172.21.15.150:0/3239921841 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9c7400bca0 con 0x7f9c84106be0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.197+0000 7f9c82ffd700 1 -- 172.21.15.150:0/3239921841 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74003d60 con 0x7f9c84106be0 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- 172.21.15.150:0/3239921841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 msgr2=0x7f9c84107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:35.178 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 --2- 172.21.15.150:0/3239921841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c74009ab0 tx=0x7f9c74009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- 172.21.15.150:0/3239921841 shutdown_connections 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 --2- 172.21.15.150:0/3239921841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c84107000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- 172.21.15.150:0/3239921841 >> 172.21.15.150:0/3239921841 conn(0x7f9c841021c0 msgr2=0x7f9c841045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- 172.21.15.150:0/3239921841 shutdown_connections 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- 172.21.15.150:0/3239921841 wait complete. 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 Processor -- start 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.198+0000 7f9c8a899700 1 -- start start 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c8a899700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c8a899700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8410b190 con 0x7f9c84106be0 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55274/0 (socket says 172.21.15.150:55274) 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c83fff700 1 -- 172.21.15.150:0/2671194074 learned_addr learned my addr 172.21.15.150:0/2671194074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c83fff700 1 -- 172.21.15.150:0/2671194074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8419c270 con 0x7f9c84106be0 2024-03-18T15:50:35.179 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c83fff700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c74003db0 tx=0x7f9c74003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c7400be50 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8419c500 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9c74004330 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c7401a5c0 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.199+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8419f1f0 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.200+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9c7401a7f0 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.200+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c840693d0 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.200+0000 7f9c817fa700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 0x7f9c6c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.200+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9c7404d080 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.200+0000 7f9c837fe700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 0x7f9c6c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.201+0000 7f9c837fe700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 0x7f9c6c03b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006fd0 tx=0x7f9c78006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.206+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c7402c7a0 con 0x7f9c84106be0 2024-03-18T15:50:35.180 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.401+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f9c840008d0 con 0x7f9c6c038ed0 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.534+0000 7f9c817fa700 1 -- 172.21.15.150:0/2671194074 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9c840008d0 con 0x7f9c6c038ed0 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 msgr2=0x7f9c6c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 0x7f9c6c03b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006fd0 tx=0x7f9c78006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 msgr2=0x7f9c8419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9c74003db0 tx=0x7f9c74003e90 comp rx=0 tx=0).stop 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 shutdown_connections 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f9c6c038ed0 0x7f9c6c03b390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 --2- 172.21.15.150:0/2671194074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c84106be0 0x7f9c8419bd30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 >> 172.21.15.150:0/2671194074 conn(0x7f9c841021c0 msgr2=0x7f9c841030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 shutdown_connections 2024-03-18T15:50:35.181 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:34.536+0000 7f9c8a899700 1 -- 172.21.15.150:0/2671194074 wait complete. 2024-03-18T15:50:36.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:35 smithi150 ceph-mon[31946]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:36.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:35 smithi150 ceph-mon[31946]: Generating ssh key... 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f8865306700 1 Processor -- start 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f8865306700 1 -- start start 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f8865306700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f8865306700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f886007c6f0 con 0x7f8860108790 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f885effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f885effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55290/0 (socket says 172.21.15.150:55290) 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.563+0000 7f885effd700 1 -- 172.21.15.150:0/3031678663 learned_addr learned my addr 172.21.15.150:0/3031678663 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:36.366 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f885effd700 1 -- 172.21.15.150:0/3031678663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886010b0c0 con 0x7f8860108790 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f885effd700 1 --2- 172.21.15.150:0/3031678663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8850009ab0 tx=0x7f8850009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d9a3ee11cce441 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f885e7fc700 1 -- 172.21.15.150:0/3031678663 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8850004160 con 0x7f8860108790 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f885e7fc700 1 -- 172.21.15.150:0/3031678663 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f885000bca0 con 0x7f8860108790 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f885e7fc700 1 -- 172.21.15.150:0/3031678663 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8850003d60 con 0x7f8860108790 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f8865306700 1 -- 172.21.15.150:0/3031678663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 msgr2=0x7f886010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f8865306700 1 --2- 172.21.15.150:0/3031678663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8850009ab0 tx=0x7f8850009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f8865306700 1 -- 172.21.15.150:0/3031678663 shutdown_connections 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f8865306700 1 --2- 172.21.15.150:0/3031678663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886010ab80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.564+0000 7f8865306700 1 -- 172.21.15.150:0/3031678663 >> 172.21.15.150:0/3031678663 conn(0x7f88601021c0 msgr2=0x7f88601045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.565+0000 7f8865306700 1 -- 172.21.15.150:0/3031678663 shutdown_connections 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.565+0000 7f8865306700 1 -- 172.21.15.150:0/3031678663 wait complete. 2024-03-18T15:50:36.367 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.565+0000 7f8865306700 1 Processor -- start 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.565+0000 7f8865306700 1 -- start start 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.565+0000 7f8865306700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f8865306700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f886010c0e0 con 0x7f8860108790 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55300/0 (socket says 172.21.15.150:55300) 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885effd700 1 -- 172.21.15.150:0/2940867294 learned_addr learned my addr 172.21.15.150:0/2940867294 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885effd700 1 -- 172.21.15.150:0/2940867294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886019c1b0 con 0x7f8860108790 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885effd700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8850003aa0 tx=0x7f8850003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f885000be50 con 0x7f8860108790 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8850004340 con 0x7f8860108790 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886019c440 con 0x7f8860108790 2024-03-18T15:50:36.368 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f885001a780 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.566+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886019f130 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.567+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8850022020 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.567+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88600693d0 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.567+0000 7f885cff9700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 0x7f884c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.567+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f885004ce50 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.567+0000 7f88567fc700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 0x7f884c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.568+0000 7f88567fc700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 0x7f884c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8848006fd0 tx=0x7f8848006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.572+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f88500044f0 con 0x7f8860108790 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.766+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8860103450 con 0x7f884c038f20 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.766+0000 7f885cff9700 1 -- 172.21.15.150:0/2940867294 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8860103450 con 0x7f884c038f20 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.768+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 msgr2=0x7f884c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:36.369 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.768+0000 7f8865306700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 0x7f884c03b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8848006fd0 tx=0x7f8848006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.768+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 msgr2=0x7f886019bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.768+0000 7f8865306700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8850003aa0 tx=0x7f8850003e60 comp rx=0 tx=0).stop 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 shutdown_connections 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f884c038f20 0x7f884c03b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 --2- 172.21.15.150:0/2940867294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8860108790 0x7f886019bc70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 >> 172.21.15.150:0/2940867294 conn(0x7f88601021c0 msgr2=0x7f8860102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 shutdown_connections 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:35.769+0000 7f8865306700 1 -- 172.21.15.150:0/2940867294 wait complete. 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T15:50:36.370 INFO:teuthology.orchestra.run.smithi150.stdout:Adding host smithi150... 2024-03-18T15:50:37.221 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:36 smithi150 ceph-mon[31946]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:38.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:37 smithi150 ceph-mon[31946]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "addr": "172.21.15.150", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:39.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:38 smithi150 ceph-mon[31946]: Deploying cephadm binary to smithi150 2024-03-18T15:50:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Added host 'smithi150' with addr '172.21.15.150' 2024-03-18T15:50:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40b2803700 1 Processor -- start 2024-03-18T15:50:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40b2803700 1 -- start start 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40b2803700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40b2803700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac1086c0 con 0x7f40ac107cd0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55310/0 (socket says 172.21.15.150:55310) 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40abfff700 1 -- 172.21.15.150:0/3104628524 learned_addr learned my addr 172.21.15.150:0/3104628524 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.785+0000 7f40abfff700 1 -- 172.21.15.150:0/3104628524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac108ed0 con 0x7f40ac107cd0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40abfff700 1 --2- 172.21.15.150:0/3104628524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f409c0098a0 tx=0x7f409c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b0a8fbf58d398d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40aaffd700 1 -- 172.21.15.150:0/3104628524 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c004160 con 0x7f40ac107cd0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40aaffd700 1 -- 172.21.15.150:0/3104628524 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f409c004300 con 0x7f40ac107cd0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40aaffd700 1 -- 172.21.15.150:0/3104628524 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c011b50 con 0x7f40ac107cd0 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 -- 172.21.15.150:0/3104628524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 msgr2=0x7f40ac1080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 --2- 172.21.15.150:0/3104628524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f409c0098a0 tx=0x7f409c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:41.808 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 -- 172.21.15.150:0/3104628524 shutdown_connections 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 --2- 172.21.15.150:0/3104628524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac1080f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 -- 172.21.15.150:0/3104628524 >> 172.21.15.150:0/3104628524 conn(0x7f40ac1032b0 msgr2=0x7f40ac1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 -- 172.21.15.150:0/3104628524 shutdown_connections 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.786+0000 7f40b2803700 1 -- 172.21.15.150:0/3104628524 wait complete. 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40b2803700 1 Processor -- start 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40b2803700 1 -- start start 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40b2803700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40b2803700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac10c280 con 0x7f40ac107cd0 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55320/0 (socket says 172.21.15.150:55320) 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.787+0000 7f40abfff700 1 -- 172.21.15.150:0/226874317 learned_addr learned my addr 172.21.15.150:0/226874317 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40abfff700 1 -- 172.21.15.150:0/226874317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac19c230 con 0x7f40ac107cd0 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40abfff700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f409c009fb0 tx=0x7f409c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:41.809 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c0038b0 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40ac19c4c0 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f409c011e50 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c019550 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.788+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40ac19f1a0 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.789+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f409c019800 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.789+0000 7f40a97fa700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 0x7f409403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.789+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f409c04d0f0 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.789+0000 7f40ab7fe700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 0x7f409403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.789+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40ac063900 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.790+0000 7f40ab7fe700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 0x7f409403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006fd0 tx=0x7f40a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.794+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f409c00f2e0 con 0x7f40ac107cd0 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:36.993+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "addr": "172.21.15.150", "target": ["mon-mgr", ""]}) -- 0x7f40ac1045a0 con 0x7f4094038f20 2024-03-18T15:50:41.810 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.214+0000 7f40a97fa700 1 -- 172.21.15.150:0/226874317 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f40ac1045a0 con 0x7f4094038f20 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 msgr2=0x7f409403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 0x7f409403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006fd0 tx=0x7f40a0006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 msgr2=0x7f40ac19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f409c009fb0 tx=0x7f409c011bf0 comp rx=0 tx=0).stop 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 shutdown_connections 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f4094038f20 0x7f409403b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 --2- 172.21.15.150:0/226874317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40ac107cd0 0x7f40ac19bcf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 >> 172.21.15.150:0/226874317 conn(0x7f40ac1032b0 msgr2=0x7f40ac103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 shutdown_connections 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:41.217+0000 7f40b2803700 1 -- 172.21.15.150:0/226874317 wait complete. 2024-03-18T15:50:41.811 INFO:teuthology.orchestra.run.smithi150.stdout:Deploying unmanaged mon service... 2024-03-18T15:50:42.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:42 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:42.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:42 smithi150 ceph-mon[31946]: Added host smithi150 2024-03-18T15:50:42.468 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:42 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:43.091 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.248+0000 7fe626d4f700 1 Processor -- start 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.248+0000 7fe626d4f700 1 -- start start 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.248+0000 7fe626d4f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.248+0000 7fe626d4f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe62007be60 con 0x7fe620108770 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe624aeb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe624aeb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55336/0 (socket says 172.21.15.150:55336) 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe624aeb700 1 -- 172.21.15.150:0/816719980 learned_addr learned my addr 172.21.15.150:0/816719980 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe624aeb700 1 -- 172.21.15.150:0/816719980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62010b0a0 con 0x7fe620108770 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe624aeb700 1 --2- 172.21.15.150:0/816719980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6140098a0 tx=0x7fe614009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f162460b10df53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe61f7fe700 1 -- 172.21.15.150:0/816719980 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe614004160 con 0x7fe620108770 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe61f7fe700 1 -- 172.21.15.150:0/816719980 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe614004300 con 0x7fe620108770 2024-03-18T15:50:43.092 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.249+0000 7fe61f7fe700 1 -- 172.21.15.150:0/816719980 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe614011b50 con 0x7fe620108770 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- 172.21.15.150:0/816719980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 msgr2=0x7fe62010ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 --2- 172.21.15.150:0/816719980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6140098a0 tx=0x7fe614009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- 172.21.15.150:0/816719980 shutdown_connections 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 --2- 172.21.15.150:0/816719980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62010ab60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- 172.21.15.150:0/816719980 >> 172.21.15.150:0/816719980 conn(0x7fe620102140 msgr2=0x7fe6201045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- 172.21.15.150:0/816719980 shutdown_connections 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- 172.21.15.150:0/816719980 wait complete. 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 Processor -- start 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- start start 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.250+0000 7fe626d4f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe62010c0c0 con 0x7fe620108770 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe624aeb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe624aeb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55342/0 (socket says 172.21.15.150:55342) 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe624aeb700 1 -- 172.21.15.150:0/2404774643 learned_addr learned my addr 172.21.15.150:0/2404774643 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:43.093 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe624aeb700 1 -- 172.21.15.150:0/2404774643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62019c190 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe624aeb700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe614011b60 tx=0x7fe614011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe614003b20 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe614003cc0 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe614024840 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62019c420 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.251+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62019f110 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.252+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe614022020 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.252+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe620063900 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.252+0000 7fe61dffb700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 0x7fe61003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.252+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe614011e00 con 0x7fe620108770 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.252+0000 7fe61ffff700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 0x7fe61003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:43.094 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.253+0000 7fe61ffff700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 0x7fe61003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe60c006fd0 tx=0x7fe60c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.257+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe61400b3e0 con 0x7fe620108770 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.467+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe620103380 con 0x7fe610038f70 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.474+0000 7fe61dffb700 1 -- 172.21.15.150:0/2404774643 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe620103380 con 0x7fe610038f70 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.476+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 msgr2=0x7fe61003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.476+0000 7fe626d4f700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 0x7fe61003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe60c006fd0 tx=0x7fe60c006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.476+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 msgr2=0x7fe62019bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.476+0000 7fe626d4f700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe614011b60 tx=0x7fe614011c40 comp rx=0 tx=0).stop 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 shutdown_connections 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fe610038f70 0x7fe61003b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 --2- 172.21.15.150:0/2404774643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe620108770 0x7fe62019bc50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 >> 172.21.15.150:0/2404774643 conn(0x7fe620102140 msgr2=0x7fe620102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 shutdown_connections 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:42.477+0000 7fe626d4f700 1 -- 172.21.15.150:0/2404774643 wait complete. 2024-03-18T15:50:43.095 INFO:teuthology.orchestra.run.smithi150.stdout:Deploying unmanaged mgr service... 2024-03-18T15:50:43.686 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:43 smithi150 ceph-mon[31946]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:43.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:43 smithi150 ceph-mon[31946]: Saving service mon spec with placement count:5 2024-03-18T15:50:43.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:43 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T15:50:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.427+0000 7f6f862a5700 1 Processor -- start 2024-03-18T15:50:44.577 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.427+0000 7f6f862a5700 1 -- start start 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f862a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f862a5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801086c0 con 0x7f6f80107cd0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42016/0 (socket says 172.21.15.150:42016) 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7ffff700 1 -- 172.21.15.150:0/1503966942 learned_addr learned my addr 172.21.15.150:0/1503966942 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7ffff700 1 -- 172.21.15.150:0/1503966942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80108ed0 con 0x7f6f80107cd0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7ffff700 1 --2- 172.21.15.150:0/1503966942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f700098a0 tx=0x7f6f70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bad86382bf3b58 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7effd700 1 -- 172.21.15.150:0/1503966942 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f70004160 con 0x7f6f80107cd0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7effd700 1 -- 172.21.15.150:0/1503966942 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f70004300 con 0x7f6f80107cd0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.428+0000 7f6f7effd700 1 -- 172.21.15.150:0/1503966942 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f70011b50 con 0x7f6f80107cd0 2024-03-18T15:50:44.578 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 -- 172.21.15.150:0/1503966942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 msgr2=0x7f6f801080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 --2- 172.21.15.150:0/1503966942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f700098a0 tx=0x7f6f70009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 -- 172.21.15.150:0/1503966942 shutdown_connections 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 --2- 172.21.15.150:0/1503966942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f801080f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 -- 172.21.15.150:0/1503966942 >> 172.21.15.150:0/1503966942 conn(0x7f6f801032b0 msgr2=0x7f6f801056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 -- 172.21.15.150:0/1503966942 shutdown_connections 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.429+0000 7f6f862a5700 1 -- 172.21.15.150:0/1503966942 wait complete. 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f862a5700 1 Processor -- start 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f862a5700 1 -- start start 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f862a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f862a5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8010c280 con 0x7f6f80107cd0 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:44.579 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42032/0 (socket says 172.21.15.150:42032) 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f7ffff700 1 -- 172.21.15.150:0/2152986021 learned_addr learned my addr 172.21.15.150:0/2152986021 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f7ffff700 1 -- 172.21.15.150:0/2152986021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8019c230 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.430+0000 7f6f7ffff700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011850 tx=0x7f6f70011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f70003af0 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f70020070 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8019c4c0 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f700195a0 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8019f1a0 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.431+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6f700198a0 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.432+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f80063900 con 0x7f6f80107cd0 2024-03-18T15:50:44.580 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.432+0000 7f6f7d7fa700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 0x7f6f6803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.432+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6f70011e00 con 0x7f6f80107cd0 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.432+0000 7f6f7f7fe700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 0x7f6f6803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.432+0000 7f6f7f7fe700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 0x7f6f6803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f74006fd0 tx=0x7f6f74006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.437+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f70023370 con 0x7f6f80107cd0 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.649+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6f801045a0 con 0x7f6f68038f20 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.656+0000 7f6f7d7fa700 1 -- 172.21.15.150:0/2152986021 <== mgr.14118 v2:172.21.15.150:6800/775411148 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6f801045a0 con 0x7f6f68038f20 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 msgr2=0x7f6f6803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 0x7f6f6803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6f74006fd0 tx=0x7f6f74006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 msgr2=0x7f6f8019bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011850 tx=0x7f6f70011c10 comp rx=0 tx=0).stop 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 shutdown_connections 2024-03-18T15:50:44.581 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f6f68038f20 0x7f6f6803b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:44.582 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 --2- 172.21.15.150:0/2152986021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f80107cd0 0x7f6f8019bcf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:44.582 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 >> 172.21.15.150:0/2152986021 conn(0x7f6f801032b0 msgr2=0x7f6f80103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:44.582 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 shutdown_connections 2024-03-18T15:50:44.582 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:43.658+0000 7f6f862a5700 1 -- 172.21.15.150:0/2152986021 wait complete. 2024-03-18T15:50:44.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:44 smithi150 ceph-mon[31946]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:44.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:44 smithi150 ceph-mon[31946]: Saving service mgr spec with placement count:2 2024-03-18T15:50:44.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:44 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12ff020700 1 Processor -- start 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12ff020700 1 -- start start 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12ff020700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12ff020700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81086e0 con 0x7f12f8107cf0 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12fcdbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12fcdbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42040/0 (socket says 172.21.15.150:42040) 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.958+0000 7f12fcdbc700 1 -- 172.21.15.150:0/2499222380 learned_addr learned my addr 172.21.15.150:0/2499222380 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12fcdbc700 1 -- 172.21.15.150:0/2499222380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f8108ef0 con 0x7f12f8107cf0 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12fcdbc700 1 --2- 172.21.15.150:0/2499222380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009ab0 tx=0x7f12ec009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80e619b101e5862 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:46.004 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12f77fe700 1 -- 172.21.15.150:0/2499222380 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12ec004160 con 0x7f12f8107cf0 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12f77fe700 1 -- 172.21.15.150:0/2499222380 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12ec00bca0 con 0x7f12f8107cf0 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12f77fe700 1 -- 172.21.15.150:0/2499222380 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12ec003d60 con 0x7f12f8107cf0 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12ff020700 1 -- 172.21.15.150:0/2499222380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 msgr2=0x7f12f8108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.959+0000 7f12ff020700 1 --2- 172.21.15.150:0/2499222380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009ab0 tx=0x7f12ec009dc0 comp rx=0 tx=0).stop 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- 172.21.15.150:0/2499222380 shutdown_connections 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 --2- 172.21.15.150:0/2499222380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f8108110 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- 172.21.15.150:0/2499222380 >> 172.21.15.150:0/2499222380 conn(0x7f12f81032b0 msgr2=0x7f12f81056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- 172.21.15.150:0/2499222380 shutdown_connections 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- 172.21.15.150:0/2499222380 wait complete. 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 Processor -- start 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- start start 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12ff020700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12f810c2a0 con 0x7f12f8107cf0 2024-03-18T15:50:46.005 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12fcdbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12fcdbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42044/0 (socket says 172.21.15.150:42044) 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.960+0000 7f12fcdbc700 1 -- 172.21.15.150:0/2883765459 learned_addr learned my addr 172.21.15.150:0/2883765459 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12fcdbc700 1 -- 172.21.15.150:0/2883765459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f819c270 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12fcdbc700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12ec003db0 tx=0x7f12ec003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12ec00be50 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12ec004330 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f819c500 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12ec01a5c0 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.961+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f819f1f0 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.962+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f12ec0119c0 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.962+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f80693d0 con 0x7f12f8107cf0 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.962+0000 7f12f5ffb700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 0x7f12e803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.962+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f12ec04d070 con 0x7f12f8107cf0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.962+0000 7f12f7fff700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 0x7f12e803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.963+0000 7f12f7fff700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 0x7f12e803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f12e4006fd0 tx=0x7f12e4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:44.967+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12ec02c3a0 con 0x7f12f8107cf0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.165+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f12f819f840 con 0x7f12f8107cf0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.168+0000 7f12f5ffb700 1 -- 172.21.15.150:0/2883765459 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f12ec02c580 con 0x7f12f8107cf0 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 msgr2=0x7f12e803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 0x7f12e803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f12e4006fd0 tx=0x7f12e4006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 msgr2=0x7f12f819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12ec003db0 tx=0x7f12ec003e90 comp rx=0 tx=0).stop 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 shutdown_connections 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f12e8038ed0 0x7f12e803b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 --2- 172.21.15.150:0/2883765459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8107cf0 0x7f12f819bd30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 >> 172.21.15.150:0/2883765459 conn(0x7f12f81032b0 msgr2=0x7f12f8103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:46.007 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 shutdown_connections 2024-03-18T15:50:46.008 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:45.171+0000 7f12ff020700 1 -- 172.21.15.150:0/2883765459 wait complete. 2024-03-18T15:50:46.299 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2883765459' entity='client.admin' 2024-03-18T15:50:46.299 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:46 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.401+0000 7fae247e4700 1 Processor -- start 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.401+0000 7fae247e4700 1 -- start start 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae247e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae247e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c106990 con 0x7fae1c105fa0 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae22580700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:47.210 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae22580700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42046/0 (socket says 172.21.15.150:42046) 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae22580700 1 -- 172.21.15.150:0/3770963560 learned_addr learned my addr 172.21.15.150:0/3770963560 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae22580700 1 -- 172.21.15.150:0/3770963560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c106b10 con 0x7fae1c105fa0 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae22580700 1 --2- 172.21.15.150:0/3770963560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae14009e80 tx=0x7fae1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42017febd8e5c6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae2157e700 1 -- 172.21.15.150:0/3770963560 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae14017070 con 0x7fae1c105fa0 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae2157e700 1 -- 172.21.15.150:0/3770963560 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae1400b8c0 con 0x7fae1c105fa0 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.402+0000 7fae2157e700 1 -- 172.21.15.150:0/3770963560 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae140045c0 con 0x7fae1c105fa0 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 -- 172.21.15.150:0/3770963560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 msgr2=0x7fae1c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 --2- 172.21.15.150:0/3770963560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae14009e80 tx=0x7fae1400b330 comp rx=0 tx=0).stop 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 -- 172.21.15.150:0/3770963560 shutdown_connections 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 --2- 172.21.15.150:0/3770963560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c1063c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 -- 172.21.15.150:0/3770963560 >> 172.21.15.150:0/3770963560 conn(0x7fae1c1018c0 msgr2=0x7fae1c103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 -- 172.21.15.150:0/3770963560 shutdown_connections 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.403+0000 7fae247e4700 1 -- 172.21.15.150:0/3770963560 wait complete. 2024-03-18T15:50:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae247e4700 1 Processor -- start 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae247e4700 1 -- start start 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae247e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae247e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c250 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae22580700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae22580700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42060/0 (socket says 172.21.15.150:42060) 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.404+0000 7fae22580700 1 -- 172.21.15.150:0/870696571 learned_addr learned my addr 172.21.15.150:0/870696571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae22580700 1 -- 172.21.15.150:0/870696571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c19c170 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae22580700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b7b0 tx=0x7fae1400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae14020020 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae14003d90 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c19c400 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fae14017040 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.405+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c19f0f0 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.406+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fae14023020 con 0x7fae1c105fa0 2024-03-18T15:50:47.212 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.406+0000 7fae137fe700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 0x7fae0803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.406+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fae1400f050 con 0x7fae1c105fa0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.406+0000 7fae21d7f700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 0x7fae0803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.406+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1c063900 con 0x7fae1c105fa0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.407+0000 7fae21d7f700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 0x7fae0803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae0c006fd0 tx=0x7fae0c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.411+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae1402a020 con 0x7fae1c105fa0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.622+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fae1c19f6b0 con 0x7fae1c105fa0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.625+0000 7fae137fe700 1 -- 172.21.15.150:0/870696571 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fae14018240 con 0x7fae1c105fa0 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 msgr2=0x7fae0803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 0x7fae0803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae0c006fd0 tx=0x7fae0c006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 msgr2=0x7fae1c19bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b7b0 tx=0x7fae1400bd50 comp rx=0 tx=0).stop 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 shutdown_connections 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fae08038ed0 0x7fae0803b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:47.213 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 --2- 172.21.15.150:0/870696571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae1c105fa0 0x7fae1c19bc30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:47.214 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 >> 172.21.15.150:0/870696571 conn(0x7fae1c1018c0 msgr2=0x7fae1c102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:47.214 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 shutdown_connections 2024-03-18T15:50:47.214 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:46.629+0000 7fae247e4700 1 -- 172.21.15.150:0/870696571 wait complete. 2024-03-18T15:50:47.214 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling the dashboard module... 2024-03-18T15:50:47.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:47 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:47.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/870696571' entity='client.admin' 2024-03-18T15:50:48.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2243344001' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T15:50:48.822 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:48 smithi150 ceph-mon[31946]: from='mgr.14118 172.21.15.150:0/56506105' entity='mgr.x' 2024-03-18T15:50:48.822 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setuser ceph since I am not root 2024-03-18T15:50:48.822 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setgroup ceph since I am not root 2024-03-18T15:50:49.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:49.069+0000 7fe709171200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d85ee4700 1 Processor -- start 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d85ee4700 1 -- start start 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d85ee4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d85ee4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8007d170 con 0x7f7d8007e740 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42074/0 (socket says 172.21.15.150:42074) 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.691+0000 7f7d7f7fe700 1 -- 172.21.15.150:0/1302027562 learned_addr learned my addr 172.21.15.150:0/1302027562 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:49.105 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.692+0000 7f7d7f7fe700 1 -- 172.21.15.150:0/1302027562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8007d2f0 con 0x7f7d8007e740 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.692+0000 7f7d7f7fe700 1 --2- 172.21.15.150:0/1302027562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d700098a0 tx=0x7f7d70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80e9462aa295e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.692+0000 7f7d7e7fc700 1 -- 172.21.15.150:0/1302027562 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d70004160 con 0x7f7d8007e740 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.692+0000 7f7d7e7fc700 1 -- 172.21.15.150:0/1302027562 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70004300 con 0x7f7d8007e740 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.692+0000 7f7d7e7fc700 1 -- 172.21.15.150:0/1302027562 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d70011b20 con 0x7f7d8007e740 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 -- 172.21.15.150:0/1302027562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 msgr2=0x7f7d8007cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/1302027562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d700098a0 tx=0x7f7d70009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 -- 172.21.15.150:0/1302027562 shutdown_connections 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/1302027562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8007cba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 -- 172.21.15.150:0/1302027562 >> 172.21.15.150:0/1302027562 conn(0x7f7d80103260 msgr2=0x7f7d801056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 -- 172.21.15.150:0/1302027562 shutdown_connections 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 -- 172.21.15.150:0/1302027562 wait complete. 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.693+0000 7f7d85ee4700 1 Processor -- start 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d85ee4700 1 -- start start 2024-03-18T15:50:49.106 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d85ee4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d85ee4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8010e300 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d7f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42078/0 (socket says 172.21.15.150:42078) 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d7f7fe700 1 -- 172.21.15.150:0/2243344001 learned_addr learned my addr 172.21.15.150:0/2243344001 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.694+0000 7f7d7f7fe700 1 -- 172.21.15.150:0/2243344001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8019e3c0 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d7f7fe700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7d700116a0 tx=0x7f7d70011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d70003a40 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70003be0 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8019e650 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8019eab0 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.695+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d7001a720 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.696+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7d7001aa50 con 0x7f7d8007e740 2024-03-18T15:50:49.107 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.696+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d80063900 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.697+0000 7f7d7cff9700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 0x7f7d6803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.697+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d7004e3d0 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.697+0000 7f7d7effd700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 0x7f7d6803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.697+0000 7f7d7effd700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 0x7f7d6803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.701+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d70011e20 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:47.963+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f7d801a1480 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.467+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f7d7000bc30 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.469+0000 7f7d7cff9700 1 -- 172.21.15.150:0/2243344001 <== mon.0 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f7d7002d910 con 0x7f7d8007e740 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 msgr2=0x7f7d6803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 0x7f7d6803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 msgr2=0x7f7d8019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7d700116a0 tx=0x7f7d70011ba0 comp rx=0 tx=0).stop 2024-03-18T15:50:49.108 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 shutdown_connections 2024-03-18T15:50:49.109 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f7d68038f80 0x7f7d6803b440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:49.109 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 --2- 172.21.15.150:0/2243344001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8007e740 0x7f7d8019de80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:49.109 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 >> 172.21.15.150:0/2243344001 conn(0x7f7d80103260 msgr2=0x7f7d80103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:49.109 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 shutdown_connections 2024-03-18T15:50:49.109 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:48.471+0000 7f7d85ee4700 1 -- 172.21.15.150:0/2243344001 wait complete. 2024-03-18T15:50:49.391 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:49.167+0000 7fe709171200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:50:49.696 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2243344001' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T15:50:49.696 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-mon[31946]: mgrmap e9: x(active, since 18s) 2024-03-18T15:50:50.071 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:49.842+0000 7fe709171200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:50:50.071 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:49.940+0000 7fe709171200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:50:50.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.086+0000 7fe709171200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:50:50.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.181+0000 7fe709171200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:50:50.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.264+0000 7fe709171200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:50:50.437 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:50.437 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T15:50:50.437 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:50:50.437 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.490+0000 7f6313aaa700 1 Processor -- start 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.490+0000 7f6313aaa700 1 -- start start 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.490+0000 7f6313aaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.490+0000 7f6313aaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f630c1075c0 con 0x7f630c106bd0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.491+0000 7f6311846700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.491+0000 7f6311846700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42108/0 (socket says 172.21.15.150:42108) 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.491+0000 7f6311846700 1 -- 172.21.15.150:0/1730136373 learned_addr learned my addr 172.21.15.150:0/1730136373 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.491+0000 7f6311846700 1 -- 172.21.15.150:0/1730136373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c107dd0 con 0x7f630c106bd0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.491+0000 7f6311846700 1 --2- 172.21.15.150:0/1730136373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63040098a0 tx=0x7f6304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5655def2e612aee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6310844700 1 -- 172.21.15.150:0/1730136373 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6304004160 con 0x7f630c106bd0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6310844700 1 -- 172.21.15.150:0/1730136373 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6304004300 con 0x7f630c106bd0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6310844700 1 -- 172.21.15.150:0/1730136373 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6304011b20 con 0x7f630c106bd0 2024-03-18T15:50:50.438 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 -- 172.21.15.150:0/1730136373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 msgr2=0x7f630c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 --2- 172.21.15.150:0/1730136373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63040098a0 tx=0x7f6304009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 -- 172.21.15.150:0/1730136373 shutdown_connections 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 --2- 172.21.15.150:0/1730136373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c106ff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 -- 172.21.15.150:0/1730136373 >> 172.21.15.150:0/1730136373 conn(0x7f630c102130 msgr2=0x7f630c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 -- 172.21.15.150:0/1730136373 shutdown_connections 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.492+0000 7f6313aaa700 1 -- 172.21.15.150:0/1730136373 wait complete. 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6313aaa700 1 Processor -- start 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6313aaa700 1 -- start start 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6313aaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6313aaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10b180 con 0x7f630c106bd0 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6311846700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6311846700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42120/0 (socket says 172.21.15.150:42120) 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.493+0000 7f6311846700 1 -- 172.21.15.150:0/489478867 learned_addr learned my addr 172.21.15.150:0/489478867 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f6311846700 1 -- 172.21.15.150:0/489478867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c19e440 con 0x7f630c106bd0 2024-03-18T15:50:50.439 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f6311846700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63040116a0 tx=0x7f6304011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6304003a40 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6304003be0 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630c19e6d0 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f630401a720 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.494+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630c19eb30 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f630401aa10 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630c0693d0 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f630affd700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 0x7f62f403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f630404d580 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f6311045700 1 -- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 msgr2=0x7f62f403b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.495+0000 7f6311045700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 0x7f62f403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.500+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6304011e20 con 0x7f630c106bd0 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.696+0000 7f6311045700 1 -- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 msgr2=0x7f62f403b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:50.440 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.696+0000 7f6311045700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 0x7f62f403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.755+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f630c062ee0 con 0x7f630c106bd0 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.756+0000 7f630affd700 1 -- 172.21.15.150:0/489478867 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f630401ae00 con 0x7f630c106bd0 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 msgr2=0x7f62f403b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 0x7f62f403b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 msgr2=0x7f630c19df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63040116a0 tx=0x7f6304011ba0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 shutdown_connections 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7f62f4038f80 0x7f62f403b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 --2- 172.21.15.150:0/489478867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f630c106bd0 0x7f630c19df00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 >> 172.21.15.150:0/489478867 conn(0x7f630c102130 msgr2=0x7f630c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 shutdown_connections 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:49.758+0000 7f6313aaa700 1 -- 172.21.15.150:0/489478867 wait complete. 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for the mgr to restart... 2024-03-18T15:50:50.441 INFO:teuthology.orchestra.run.smithi150.stdout:Waiting for mgr epoch 9... 2024-03-18T15:50:50.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/489478867' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:50:50.688 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.533+0000 7fe709171200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:50:50.688 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.642+0000 7fe709171200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:51.011 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.731+0000 7fe709171200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:50:51.012 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:50.831+0000 7fe709171200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:50:51.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:51.011+0000 7fe709171200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:50:52.292 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:51.963+0000 7fe709171200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:50:52.293 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:52.091+0000 7fe709171200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:50:52.591 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:52.422+0000 7fe709171200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:50:52.591 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:52.511+0000 7fe709171200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:50:53.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:52.707+0000 7fe709171200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:53.091 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:52.891+0000 7fe709171200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:50:53.431 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:53.174+0000 7fe709171200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:50:53.432 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:53.431+0000 7fe709171200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:50:53.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:53.667+0000 7fe709171200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:50:54.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:54 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:54.465+0000 7fe709171200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:50:55.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:55.032+0000 7fe709171200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:50:55.341 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:55.130+0000 7fe709171200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:50:55.841 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:50:55.346+0000 7fe709171200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:50:55.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: Active manager daemon x restarted 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: Activating manager daemon x 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: mgrmap e10: x(active, starting, since 0.00359121s) 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: Manager daemon x is now available 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:50:56.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:50:56.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:50:56.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:56 smithi150 ceph-mon[31946]: mgrmap e11: x(active, since 1.00618s) 2024-03-18T15:50:57.135 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:50:57.135 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T15:50:57.135 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:50:57.135 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:50:57.135 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde8085f700 1 Processor -- start 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde8085f700 1 -- start start 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde8085f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde8085f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c1069b0 con 0x7fde7c105fc0 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42124/0 (socket says 172.21.15.150:42124) 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.884+0000 7fde7a59c700 1 -- 172.21.15.150:0/712515775 learned_addr learned my addr 172.21.15.150:0/712515775 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.885+0000 7fde7a59c700 1 -- 172.21.15.150:0/712515775 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7c108f50 con 0x7fde7c105fc0 2024-03-18T15:50:57.136 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.885+0000 7fde7a59c700 1 --2- 172.21.15.150:0/712515775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0098a0 tx=0x7fde6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dda9bab5f087c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.885+0000 7fde7959a700 1 -- 172.21.15.150:0/712515775 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde6c004160 con 0x7fde7c105fc0 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.885+0000 7fde7959a700 1 -- 172.21.15.150:0/712515775 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde6c004300 con 0x7fde7c105fc0 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.885+0000 7fde7959a700 1 -- 172.21.15.150:0/712515775 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde6c011b20 con 0x7fde7c105fc0 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 -- 172.21.15.150:0/712515775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 msgr2=0x7fde7c1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 --2- 172.21.15.150:0/712515775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0098a0 tx=0x7fde6c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 -- 172.21.15.150:0/712515775 shutdown_connections 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 --2- 172.21.15.150:0/712515775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c1063e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 -- 172.21.15.150:0/712515775 >> 172.21.15.150:0/712515775 conn(0x7fde7c1018b0 msgr2=0x7fde7c103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 -- 172.21.15.150:0/712515775 shutdown_connections 2024-03-18T15:50:57.137 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 -- 172.21.15.150:0/712515775 wait complete. 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.886+0000 7fde8085f700 1 Processor -- start 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde8085f700 1 -- start start 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde8085f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde8085f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c10c300 con 0x7fde7c105fc0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42132/0 (socket says 172.21.15.150:42132) 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde7a59c700 1 -- 172.21.15.150:0/4112537732 learned_addr learned my addr 172.21.15.150:0/4112537732 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde7a59c700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7c19c1e0 con 0x7fde7c105fc0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde7a59c700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0116a0 tx=0x7fde6c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde6c003a40 con 0x7fde7c105fc0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde6c003be0 con 0x7fde7c105fc0 2024-03-18T15:50:57.138 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7c19c470 con 0x7fde7c105fc0 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.887+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde6c01a720 con 0x7fde7c105fc0 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.888+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7c19c8d0 con 0x7fde7c105fc0 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.888+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fde6c01aa20 con 0x7fde7c105fc0 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.889+0000 7fde6b7fe700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.889+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fde7c0693d0 con 0x7fde64038f20 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.889+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde6c04e460 con 0x7fde7c105fc0 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.889+0000 7fde79d9b700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:50.889+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:51.090+0000 7fde79d9b700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:51.090+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:51.491+0000 7fde79d9b700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:57.139 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:51.491+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:52.292+0000 7fde79d9b700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:52.292+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:53.893+0000 7fde79d9b700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/775411148 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:53.893+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:55.361+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7fde6c02d910 con 0x7fde7c105fc0 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:55.361+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 msgr2=0x7fde6403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:55.361+0000 7fde6b7fe700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.363+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fde6c02d3f0 con 0x7fde7c105fc0 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.363+0000 7fde6b7fe700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 0x7fde6403eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.363+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fde6c03cbf0 con 0x7fde6403c7b0 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.363+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 0x7fde6403eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.364+0000 7fde79d9b700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 0x7fde6403eba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fde70003a10 tx=0x7fde700092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.364+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fde6c03cbf0 con 0x7fde6403c7b0 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.371+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fde7c102b80 con 0x7fde6403c7b0 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.371+0000 7fde6b7fe700 1 -- 172.21.15.150:0/4112537732 <== mgr.14150 v2:172.21.15.150:6800/3095497318 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fde7c102b80 con 0x7fde6403c7b0 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 msgr2=0x7fde6403eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 0x7fde6403eba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fde70003a10 tx=0x7fde700092b0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 msgr2=0x7fde7c19bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0116a0 tx=0x7fde6c011ba0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 shutdown_connections 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde6403c7b0 0x7fde6403eba0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde7c105fc0 0x7fde7c19bca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 --2- 172.21.15.150:0/4112537732 >> [v2:172.21.15.150:6800/775411148,v1:172.21.15.150:6801/775411148] conn(0x7fde64038f20 0x7fde6403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:57.141 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 >> 172.21.15.150:0/4112537732 conn(0x7fde7c1018b0 msgr2=0x7fde7c102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:57.142 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 shutdown_connections 2024-03-18T15:50:57.142 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:56.372+0000 7fde8085f700 1 -- 172.21.15.150:0/4112537732 wait complete. 2024-03-18T15:50:57.142 INFO:teuthology.orchestra.run.smithi150.stdout:mgr epoch 9 is available 2024-03-18T15:50:57.142 INFO:teuthology.orchestra.run.smithi150.stdout:Generating a dashboard self-signed certificate... 2024-03-18T15:50:57.550 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:55] ENGINE Bus STARTING 2024-03-18T15:50:57.550 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:55] ENGINE Serving on http://172.21.15.150:8765 2024-03-18T15:50:57.550 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:55] ENGINE Serving on https://172.21.15.150:7150 2024-03-18T15:50:57.550 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: [18/Mar/2024:15:50:55] ENGINE Bus STARTED 2024-03-18T15:50:57.550 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:50:57.551 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:50:57.551 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.568+0000 7fdf69911700 1 Processor -- start 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.568+0000 7fdf69911700 1 -- start start 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.568+0000 7fdf69911700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.568+0000 7fdf69911700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdf641075b0 con 0x7fdf64106bc0 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38152/0 (socket says 172.21.15.150:38152) 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf62ffd700 1 -- 172.21.15.150:0/2804177475 learned_addr learned my addr 172.21.15.150:0/2804177475 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf62ffd700 1 -- 172.21.15.150:0/2804177475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64107dc0 con 0x7fdf64106bc0 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf62ffd700 1 --2- 172.21.15.150:0/2804177475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf580098a0 tx=0x7fdf58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b420ff3ef429cb2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf61ffb700 1 -- 172.21.15.150:0/2804177475 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf58004160 con 0x7fdf64106bc0 2024-03-18T15:50:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf61ffb700 1 -- 172.21.15.150:0/2804177475 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf58004300 con 0x7fdf64106bc0 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.569+0000 7fdf61ffb700 1 -- 172.21.15.150:0/2804177475 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf58011b20 con 0x7fdf64106bc0 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 -- 172.21.15.150:0/2804177475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 msgr2=0x7fdf64106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 --2- 172.21.15.150:0/2804177475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf580098a0 tx=0x7fdf58009bb0 comp rx=0 tx=0).stop 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 -- 172.21.15.150:0/2804177475 shutdown_connections 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 --2- 172.21.15.150:0/2804177475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf64106fe0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 -- 172.21.15.150:0/2804177475 >> 172.21.15.150:0/2804177475 conn(0x7fdf64102160 msgr2=0x7fdf641045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 -- 172.21.15.150:0/2804177475 shutdown_connections 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.570+0000 7fdf69911700 1 -- 172.21.15.150:0/2804177475 wait complete. 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf69911700 1 Processor -- start 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf69911700 1 -- start start 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf69911700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf69911700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6407b850 con 0x7fdf64106bc0 2024-03-18T15:50:58.463 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf62ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38156/0 (socket says 172.21.15.150:38156) 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.571+0000 7fdf62ffd700 1 -- 172.21.15.150:0/2529244377 learned_addr learned my addr 172.21.15.150:0/2529244377 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf62ffd700 1 -- 172.21.15.150:0/2529244377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6410cfc0 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf62ffd700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdf580116a0 tx=0x7fdf58011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf58003a40 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf58003be0 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6410b1b0 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf5801a720 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.572+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6410b670 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.573+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fdf5801aa20 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.573+0000 7fdf6890f700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 0x7fdf4c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.573+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdf5804d3e0 con 0x7fdf64106bc0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.573+0000 7fdf627fc700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 0x7fdf4c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:50:58.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.573+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf640693d0 con 0x7fdf64106bc0 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.574+0000 7fdf627fc700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 0x7fdf4c03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf54006fd0 tx=0x7fdf54006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.578+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf58011e20 con 0x7fdf64106bc0 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.682+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fdf5802d910 con 0x7fdf64106bc0 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.817+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdf641952b0 con 0x7fdf4c038e10 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.920+0000 7fdf6890f700 1 -- 172.21.15.150:0/2529244377 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fdf641952b0 con 0x7fdf4c038e10 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 msgr2=0x7fdf4c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 0x7fdf4c03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf54006fd0 tx=0x7fdf54006e40 comp rx=0 tx=0).stop 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 msgr2=0x7fdf6410ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdf580116a0 tx=0x7fdf58011ba0 comp rx=0 tx=0).stop 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 shutdown_connections 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fdf4c038e10 0x7fdf4c03b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 --2- 172.21.15.150:0/2529244377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf64106bc0 0x7fdf6410ca80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 >> 172.21.15.150:0/2529244377 conn(0x7fdf64102160 msgr2=0x7fdf641029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 shutdown_connections 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:57.923+0000 7fdf69911700 1 -- 172.21.15.150:0/2529244377 wait complete. 2024-03-18T15:50:58.465 INFO:teuthology.orchestra.run.smithi150.stdout:Creating initial admin user... 2024-03-18T15:50:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:58 smithi150 ceph-mon[31946]: mgrmap e12: x(active, since 2s) 2024-03-18T15:50:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:58 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:50:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:58 smithi150 ceph-mon[31946]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:50:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:58 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:50:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:50:58 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:00.508 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3JJ3xaHexncOR/4LASug.u1oGMb/vHLJqMRSAXynVvo2TNovnyPtO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710777059, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T15:51:00.508 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff74de4d700 1 Processor -- start 2024-03-18T15:51:00.508 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff74de4d700 1 -- start start 2024-03-18T15:51:00.508 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff74de4d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff74de4d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff74807d130 con 0x7ff74807e700 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff7477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff7477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38168/0 (socket says 172.21.15.150:38168) 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff7477fe700 1 -- 172.21.15.150:0/2751004194 learned_addr learned my addr 172.21.15.150:0/2751004194 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.892+0000 7ff7477fe700 1 -- 172.21.15.150:0/2751004194 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74807d2b0 con 0x7ff74807e700 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff7477fe700 1 --2- 172.21.15.150:0/2751004194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff738009ab0 tx=0x7ff738009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecc3bffc9c0bc9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff7467fc700 1 -- 172.21.15.150:0/2751004194 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff738004160 con 0x7ff74807e700 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff7467fc700 1 -- 172.21.15.150:0/2751004194 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff73800bcc0 con 0x7ff74807e700 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff7467fc700 1 -- 172.21.15.150:0/2751004194 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff738003d00 con 0x7ff74807e700 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 -- 172.21.15.150:0/2751004194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 msgr2=0x7ff74807cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 --2- 172.21.15.150:0/2751004194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff738009ab0 tx=0x7ff738009dc0 comp rx=0 tx=0).stop 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 -- 172.21.15.150:0/2751004194 shutdown_connections 2024-03-18T15:51:00.509 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 --2- 172.21.15.150:0/2751004194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74807cb60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 -- 172.21.15.150:0/2751004194 >> 172.21.15.150:0/2751004194 conn(0x7ff7481032b0 msgr2=0x7ff7481056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 -- 172.21.15.150:0/2751004194 shutdown_connections 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.893+0000 7ff74de4d700 1 -- 172.21.15.150:0/2751004194 wait complete. 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.894+0000 7ff74de4d700 1 Processor -- start 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.894+0000 7ff74de4d700 1 -- start start 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.894+0000 7ff74de4d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.894+0000 7ff74de4d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff74810e370 con 0x7ff74807e700 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.894+0000 7ff7477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff7477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38184/0 (socket says 172.21.15.150:38184) 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff7477fe700 1 -- 172.21.15.150:0/3985830691 learned_addr learned my addr 172.21.15.150:0/3985830691 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff7477fe700 1 -- 172.21.15.150:0/3985830691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74819e440 con 0x7ff74807e700 2024-03-18T15:51:00.510 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff7477fe700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff738003a40 tx=0x7ff73800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff738004300 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7380044a0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74819e6d0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff738011740 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.895+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7481a13c0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.896+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7ff738022020 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.896+0000 7ff744ff9700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 0x7ff73003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.896+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7480693d0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.896+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff73804cff0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.896+0000 7ff746ffd700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 0x7ff73003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.897+0000 7ff746ffd700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 0x7ff73003b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff73c006fd0 tx=0x7ff73c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:58.901+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7380118e0 con 0x7ff74807e700 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.134+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff7481a1650 con 0x7ff730038f80 2024-03-18T15:51:00.511 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.360+0000 7ff744ff9700 1 -- 172.21.15.150:0/3985830691 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff7481a1650 con 0x7ff730038f80 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 msgr2=0x7ff73003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 0x7ff73003b440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff73c006fd0 tx=0x7ff73c006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 msgr2=0x7ff74819df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff738003a40 tx=0x7ff73800bac0 comp rx=0 tx=0).stop 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 shutdown_connections 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff730038f80 0x7ff73003b440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 --2- 172.21.15.150:0/3985830691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff74807e700 0x7ff74819df00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 >> 172.21.15.150:0/3985830691 conn(0x7ff7481032b0 msgr2=0x7ff748103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 shutdown_connections 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:50:59.363+0000 7ff74de4d700 1 -- 172.21.15.150:0/3985830691 wait complete. 2024-03-18T15:51:00.512 INFO:teuthology.orchestra.run.smithi150.stdout:Fetching dashboard port number... 2024-03-18T15:51:00.629 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:00 smithi150 ceph-mon[31946]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:00.629 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:01.690 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:01 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2656219064' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T15:51:01.697 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T15:51:01.697 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.921+0000 7f0ff211a700 1 Processor -- start 2024-03-18T15:51:01.697 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.921+0000 7f0ff211a700 1 -- start start 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0ff211a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0ff211a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec07be20 con 0x7f0fec1087b0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38186/0 (socket says 172.21.15.150:38186) 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0feb7fe700 1 -- 172.21.15.150:0/1116547905 learned_addr learned my addr 172.21.15.150:0/1116547905 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0feb7fe700 1 -- 172.21.15.150:0/1116547905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec10b0e0 con 0x7f0fec1087b0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0feb7fe700 1 --2- 172.21.15.150:0/1116547905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0098a0 tx=0x7f0fdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efae88d01879c41d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0fea7fc700 1 -- 172.21.15.150:0/1116547905 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fdc004160 con 0x7f0fec1087b0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0fea7fc700 1 -- 172.21.15.150:0/1116547905 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fdc004300 con 0x7f0fec1087b0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.922+0000 7f0fea7fc700 1 -- 172.21.15.150:0/1116547905 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fdc011b20 con 0x7f0fec1087b0 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 -- 172.21.15.150:0/1116547905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 msgr2=0x7f0fec10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 --2- 172.21.15.150:0/1116547905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0098a0 tx=0x7f0fdc009bb0 comp rx=0 tx=0).stop 2024-03-18T15:51:01.698 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 -- 172.21.15.150:0/1116547905 shutdown_connections 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 --2- 172.21.15.150:0/1116547905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec10aba0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 -- 172.21.15.150:0/1116547905 >> 172.21.15.150:0/1116547905 conn(0x7f0fec102140 msgr2=0x7f0fec1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 -- 172.21.15.150:0/1116547905 shutdown_connections 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 -- 172.21.15.150:0/1116547905 wait complete. 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.923+0000 7f0ff211a700 1 Processor -- start 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0ff211a700 1 -- start start 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0ff211a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0ff211a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec10c0e0 con 0x7f0fec1087b0 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38192/0 (socket says 172.21.15.150:38192) 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0feb7fe700 1 -- 172.21.15.150:0/2656219064 learned_addr learned my addr 172.21.15.150:0/2656219064 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0feb7fe700 1 -- 172.21.15.150:0/2656219064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec19c140 con 0x7f0fec1087b0 2024-03-18T15:51:01.699 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0feb7fe700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0117e0 tx=0x7f0fdc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.924+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fdc003bf0 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.925+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fdc01a910 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.925+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fec19c3d0 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.925+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fdc024970 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.925+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fec19c830 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.925+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f0fdc022020 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.926+0000 7f0fe8ff9700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 0x7f0fd403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.926+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fec063900 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.926+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0fdc04d2f0 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.926+0000 7f0feaffd700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 0x7f0fd403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.926+0000 7f0feaffd700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 0x7f0fd403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0006fd0 tx=0x7f0fe0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:00.931+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0fdc011e20 con 0x7f0fec1087b0 2024-03-18T15:51:01.700 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.123+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f0fec19f200 con 0x7f0fec1087b0 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.123+0000 7f0fe8ff9700 1 -- 172.21.15.150:0/2656219064 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f0fdc00ba00 con 0x7f0fec1087b0 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.125+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 msgr2=0x7f0fd403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.125+0000 7f0ff211a700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 0x7f0fd403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0006fd0 tx=0x7f0fe0006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.125+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 msgr2=0x7f0fec19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.125+0000 7f0ff211a700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0117e0 tx=0x7f0fdc011c10 comp rx=0 tx=0).stop 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 shutdown_connections 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0fd4038f20 0x7f0fd403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 --2- 172.21.15.150:0/2656219064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fec1087b0 0x7f0fec19bc00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 >> 172.21.15.150:0/2656219064 conn(0x7f0fec102140 msgr2=0x7f0fec102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 shutdown_connections 2024-03-18T15:51:01.701 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:01.126+0000 7f0ff211a700 1 -- 172.21.15.150:0/2656219064 wait complete. 2024-03-18T15:51:01.705 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:51:01.705 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout disabled 2024-03-18T15:51:01.712 INFO:teuthology.orchestra.run.smithi150.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:51:01.712 INFO:teuthology.orchestra.run.smithi150.stdout:systemctl: stdout inactive 2024-03-18T15:51:01.712 INFO:teuthology.orchestra.run.smithi150.stdout:firewalld.service is not enabled 2024-03-18T15:51:01.713 INFO:teuthology.orchestra.run.smithi150.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T15:51:01.714 INFO:teuthology.orchestra.run.smithi150.stdout:Ceph Dashboard is now available at: 2024-03-18T15:51:01.714 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout: URL: https://smithi150.front.sepia.ceph.com:8443/ 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout: User: admin 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout: Password: nhko8mb0dw 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout:Saving cluster configuration to /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config directory 2024-03-18T15:51:01.715 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling autotune for osd_memory_target 2024-03-18T15:51:02.879 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a6aab700 1 Processor -- start 2024-03-18T15:51:02.879 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a6aab700 1 -- start start 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a6aab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a6aab700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0106a40 con 0x7f76a0106050 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a4847700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a4847700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38204/0 (socket says 172.21.15.150:38204) 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.144+0000 7f76a4847700 1 -- 172.21.15.150:0/3195839123 learned_addr learned my addr 172.21.15.150:0/3195839123 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f76a4847700 1 -- 172.21.15.150:0/3195839123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a0108ff0 con 0x7f76a0106050 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f76a4847700 1 --2- 172.21.15.150:0/3195839123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7698009e80 tx=0x7f769800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70753aa47f20d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f769f7fe700 1 -- 172.21.15.150:0/3195839123 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7698017070 con 0x7f76a0106050 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f769f7fe700 1 -- 172.21.15.150:0/3195839123 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f769800b8c0 con 0x7f76a0106050 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f769f7fe700 1 -- 172.21.15.150:0/3195839123 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76980045c0 con 0x7f76a0106050 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.145+0000 7f76a6aab700 1 -- 172.21.15.150:0/3195839123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 msgr2=0x7f76a0106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3195839123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7698009e80 tx=0x7f769800b330 comp rx=0 tx=0).stop 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- 172.21.15.150:0/3195839123 shutdown_connections 2024-03-18T15:51:02.880 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3195839123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a0106470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- 172.21.15.150:0/3195839123 >> 172.21.15.150:0/3195839123 conn(0x7f76a0101930 msgr2=0x7f76a0103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- 172.21.15.150:0/3195839123 shutdown_connections 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- 172.21.15.150:0/3195839123 wait complete. 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 Processor -- start 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- start start 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.146+0000 7f76a6aab700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f76a010c3a0 con 0x7f76a0106050 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a4847700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a4847700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38212/0 (socket says 172.21.15.150:38212) 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a4847700 1 -- 172.21.15.150:0/3520655378 learned_addr learned my addr 172.21.15.150:0/3520655378 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a4847700 1 -- 172.21.15.150:0/3520655378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a019c1d0 con 0x7f76a0106050 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a4847700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7698009530 tx=0x7f7698003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7698020020 con 0x7f76a0106050 2024-03-18T15:51:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f769800bba0 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a019c460 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7698017040 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.147+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a019f150 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.148+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f7698023020 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.148+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a0063900 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.149+0000 7f769dffb700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 0x7f768c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.149+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f769800f050 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.149+0000 7f769ffff700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 0x7f768c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.149+0000 7f769ffff700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 0x7f768c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f7690006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.153+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f769802a020 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.347+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f76a019f3c0 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.347+0000 7f769dffb700 1 -- 172.21.15.150:0/3520655378 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f7698007310 con 0x7f76a0106050 2024-03-18T15:51:02.882 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 msgr2=0x7f768c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 0x7f768c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f7690006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 msgr2=0x7f76a019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7698009530 tx=0x7f7698003d00 comp rx=0 tx=0).stop 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 shutdown_connections 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f768c038fd0 0x7f768c03b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 --2- 172.21.15.150:0/3520655378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76a0106050 0x7f76a019bc90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 >> 172.21.15.150:0/3520655378 conn(0x7f76a0101930 msgr2=0x7f76a0102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 shutdown_connections 2024-03-18T15:51:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:02.350+0000 7f76a6aab700 1 -- 172.21.15.150:0/3520655378 wait complete. 2024-03-18T15:51:03.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:03 smithi150 ceph-mon[31946]: mgrmap e13: x(active, since 7s) 2024-03-18T15:51:04.153 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.299+0000 7fb0dc6e2700 1 Processor -- start 2024-03-18T15:51:04.153 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0dc6e2700 1 -- start start 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0dc6e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0dc6e2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d41075b0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0da47e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0da47e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60650/0 (socket says 172.21.15.150:60650) 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0da47e700 1 -- 172.21.15.150:0/1705860229 learned_addr learned my addr 172.21.15.150:0/1705860229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.300+0000 7fb0da47e700 1 -- 172.21.15.150:0/1705860229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d4107dc0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0da47e700 1 --2- 172.21.15.150:0/1705860229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00098a0 tx=0x7fb0d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1acc7f0a64cc4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0d947c700 1 -- 172.21.15.150:0/1705860229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d0004160 con 0x7fb0d4106bc0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0d947c700 1 -- 172.21.15.150:0/1705860229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d0004300 con 0x7fb0d4106bc0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0d947c700 1 -- 172.21.15.150:0/1705860229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d0011b20 con 0x7fb0d4106bc0 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/1705860229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 msgr2=0x7fb0d4106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/1705860229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00098a0 tx=0x7fb0d0009bb0 comp rx=0 tx=0).stop 2024-03-18T15:51:04.154 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/1705860229 shutdown_connections 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/1705860229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d4106fe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/1705860229 >> 172.21.15.150:0/1705860229 conn(0x7fb0d4102160 msgr2=0x7fb0d41045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/1705860229 shutdown_connections 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.301+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/1705860229 wait complete. 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0dc6e2700 1 Processor -- start 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0dc6e2700 1 -- start start 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0dc6e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0dc6e2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d410b170 con 0x7fb0d4106bc0 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0da47e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0da47e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60658/0 (socket says 172.21.15.150:60658) 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.302+0000 7fb0da47e700 1 -- 172.21.15.150:0/3096730531 learned_addr learned my addr 172.21.15.150:0/3096730531 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0da47e700 1 -- 172.21.15.150:0/3096730531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d419c1b0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0da47e700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0000c00 tx=0x7fb0d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d0003b60 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d001a910 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d419c440 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0d0024930 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.303+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d419c8a0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fb0d0022020 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d4063900 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0cb7fe700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 0x7fb0c003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb0d004d220 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0d9c7d700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 0x7fb0c003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.304+0000 7fb0d9c7d700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 0x7fb0c003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4006fd0 tx=0x7fb0c4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.309+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0d002d3f0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.615+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fb0d4062ee0 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.618+0000 7fb0cb7fe700 1 -- 172.21.15.150:0/3096730531 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fb0d001a430 con 0x7fb0d4106bc0 2024-03-18T15:51:04.156 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 msgr2=0x7fb0c003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 0x7fb0c003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4006fd0 tx=0x7fb0c4006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 msgr2=0x7fb0d419bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0000c00 tx=0x7fb0d0011c10 comp rx=0 tx=0).stop 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 shutdown_connections 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb0c0039070 0x7fb0c003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 --2- 172.21.15.150:0/3096730531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d4106bc0 0x7fb0d419bc70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 >> 172.21.15.150:0/3096730531 conn(0x7fb0d4102160 msgr2=0x7fb0d4103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 shutdown_connections 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr 2024-03-18T15:51:03.620+0000 7fb0dc6e2700 1 -- 172.21.15.150:0/3096730531 wait complete. 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: ceph telemetry on 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:For more information see: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:04.158 INFO:teuthology.orchestra.run.smithi150.stdout:Bootstrap complete. 2024-03-18T15:51:04.194 INFO:tasks.cephadm:Fetching config... 2024-03-18T15:51:04.194 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:04.194 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T15:51:04.209 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T15:51:04.209 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:04.209 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T15:51:04.265 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T15:51:04.265 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:04.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T15:51:04.333 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T15:51:04.333 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:04.333 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T15:51:04.388 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T15:51:04.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:51:04.479 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:04.493 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:51:04.543 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:04.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:51:04.604 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCRjWH2c3EWNRwVitu5qJA6H7L7sNnshoRUQf4V4uSK+zpOMxXIcsXvpq+G5lHPnDOim029ztBIoYH2/Ms0bzbw6Qa1CLl8MLUVfKM2TVb2WGyImYqpkEWvht1G07v/TXAd1pfRKT+x/a5Ql6Ttk9tdnuOad3wWoz8Yu2w3Ixww0mGfFCUT308uylbYajVZVyvOJE+xLGKaS3KniiGECHWxKUVLfRkXZnPXMrBwvvaRrQUXBAg6eBYeF8sD0J3v5itlBIZSQewkNwFvA1/wh9rX5oSbVYDXs5YVlHb1tzpXg7my2lB5cvjyFOMI/9uelCUuepqmmYXXk/sun5G6d+4r0Dv1YedGQDLQIiucxen+p2yp8Da8akmW06K6R4Vt6EzWcO2q2qHHs9Ll+d4yoj9Tebo+1+Ny+ubOH5hZzv1LEy/aWb7zkP7KndagWsQfGLdgP9RneRdG1KSyf29lzU/ZJGtezHfVn+m3ibNP+zs5q3vkNA25H0JT6sY/oPgt20= ceph-0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:04.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T15:51:04.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:04 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3096730531' entity='client.admin' 2024-03-18T15:51:04.859 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:07.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 -- 172.21.15.150:0/3567343245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 msgr2=0x7f59d4104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:07.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3567343245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d4104320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59c4009910 tx=0x7f59c4009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:07.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 -- 172.21.15.150:0/3567343245 shutdown_connections 2024-03-18T15:51:07.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3567343245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d4104320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:07.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 -- 172.21.15.150:0/3567343245 >> 172.21.15.150:0/3567343245 conn(0x7f59d40ff4a0 msgr2=0x7f59d41018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 -- 172.21.15.150:0/3567343245 shutdown_connections 2024-03-18T15:51:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.001+0000 7f59da2d5700 1 -- 172.21.15.150:0/3567343245 wait complete. 2024-03-18T15:51:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59da2d5700 1 Processor -- start 2024-03-18T15:51:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59da2d5700 1 -- start start 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59da2d5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59da2d5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59d4109f10 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59d3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59d3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60674/0 (socket says 172.21.15.150:60674) 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59d3fff700 1 -- 172.21.15.150:0/3617265535 learned_addr learned my addr 172.21.15.150:0/3617265535 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59d3fff700 1 -- 172.21.15.150:0/3617265535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d4199b20 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.002+0000 7f59d3fff700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59c4004fa0 tx=0x7f59c4005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c4027070 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d4199db0 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c400be40 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c400f460 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d419caa0 con 0x7f59d4103f00 2024-03-18T15:51:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.003+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f59c402a070 con 0x7f59d4103f00 2024-03-18T15:51:07.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.004+0000 7f59d17fa700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 0x7f59bc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:07.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.004+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f59c401e030 con 0x7f59d4103f00 2024-03-18T15:51:07.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.004+0000 7f59d37fe700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 0x7f59bc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:07.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.004+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d410a690 con 0x7f59d4103f00 2024-03-18T15:51:07.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.004+0000 7f59d37fe700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 0x7f59bc03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006fd0 tx=0x7f59c8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:07.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.009+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59c4013070 con 0x7f59d4103f00 2024-03-18T15:51:07.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.203+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f59d402db70 con 0x7f59d4103f00 2024-03-18T15:51:07.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.206+0000 7f59d17fa700 1 -- 172.21.15.150:0/3617265535 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f59c400fb20 con 0x7f59d4103f00 2024-03-18T15:51:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.208+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 msgr2=0x7f59bc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.208+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 0x7f59bc03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006fd0 tx=0x7f59c8006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.208+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 msgr2=0x7f59d41995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.208+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59c4004fa0 tx=0x7f59c4005080 comp rx=0 tx=0).stop 2024-03-18T15:51:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 shutdown_connections 2024-03-18T15:51:07.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59bc038fc0 0x7f59bc03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:07.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 --2- 172.21.15.150:0/3617265535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59d4103f00 0x7f59d41995e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:07.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 >> 172.21.15.150:0/3617265535 conn(0x7f59d40ff4a0 msgr2=0x7f59d40ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:07.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 shutdown_connections 2024-03-18T15:51:07.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:07.209+0000 7f59da2d5700 1 -- 172.21.15.150:0/3617265535 wait complete. 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3617265535' entity='client.admin' 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:07.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:07.837 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T15:51:07.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T15:51:08.070 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:09.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 -- 172.21.15.150:0/1328615989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 msgr2=0x7f2cbc101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/1328615989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc101030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009910 tx=0x7f2cac009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 -- 172.21.15.150:0/1328615989 shutdown_connections 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/1328615989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc101030 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 -- 172.21.15.150:0/1328615989 >> 172.21.15.150:0/1328615989 conn(0x7f2cbc0fc1d0 msgr2=0x7f2cbc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 -- 172.21.15.150:0/1328615989 shutdown_connections 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.507+0000 7f2cc0e38700 1 -- 172.21.15.150:0/1328615989 wait complete. 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cc0e38700 1 Processor -- start 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cc0e38700 1 -- start start 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cc0e38700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cc0e38700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2cbc106ca0 con 0x7f2cbc100c10 2024-03-18T15:51:09.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cba59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cba59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60700/0 (socket says 172.21.15.150:60700) 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cba59c700 1 -- 172.21.15.150:0/4049648434 learned_addr learned my addr 172.21.15.150:0/4049648434 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.508+0000 7f2cba59c700 1 -- 172.21.15.150:0/4049648434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cbc1968f0 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cba59c700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009fd0 tx=0x7f2cac005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cac027070 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cac00be40 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cbc196b80 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cac00f460 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.509+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cbc199870 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f2cac02a070 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cab7fe700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 0x7f2ca403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cbc107420 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2cac01e030 con 0x7f2cbc100c10 2024-03-18T15:51:09.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cb9d9b700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 0x7f2ca403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:09.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.510+0000 7f2cb9d9b700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 0x7f2ca403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0006fd0 tx=0x7f2cb0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:09.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.515+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cac013070 con 0x7f2cbc100c10 2024-03-18T15:51:09.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.716+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f2cbc02dc10 con 0x7f2ca4038fc0 2024-03-18T15:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.719+0000 7f2cab7fe700 1 -- 172.21.15.150:0/4049648434 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2cbc02dc10 con 0x7f2ca4038fc0 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 msgr2=0x7f2ca403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 0x7f2ca403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0006fd0 tx=0x7f2cb0006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 msgr2=0x7f2cbc1963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009fd0 tx=0x7f2cac005050 comp rx=0 tx=0).stop 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 shutdown_connections 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2ca4038fc0 0x7f2ca403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 --2- 172.21.15.150:0/4049648434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc100c10 0x7f2cbc1963b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 >> 172.21.15.150:0/4049648434 conn(0x7f2cbc0fc1d0 msgr2=0x7f2cbc0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 shutdown_connections 2024-03-18T15:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:09.721+0000 7f2cc0e38700 1 -- 172.21.15.150:0/4049648434 wait complete. 2024-03-18T15:51:10.463 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2024-03-18T15:51:10.463 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:51:10.463 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:51:10.481 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:51:10.481 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:10.538 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2024-03-18T15:51:10.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch host add smithi156 2024-03-18T15:51:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:51:10.980 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:10.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:10.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 -- 172.21.15.150:0/2821253854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 msgr2=0x7f0bfc1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 --2- 172.21.15.150:0/2821253854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc1022d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009910 tx=0x7f0bec009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 -- 172.21.15.150:0/2821253854 shutdown_connections 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 --2- 172.21.15.150:0/2821253854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc1022d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 -- 172.21.15.150:0/2821253854 >> 172.21.15.150:0/2821253854 conn(0x7f0bfc0fd7a0 msgr2=0x7f0bfc0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 -- 172.21.15.150:0/2821253854 shutdown_connections 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.168+0000 7f0c016e4700 1 -- 172.21.15.150:0/2821253854 wait complete. 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0c016e4700 1 Processor -- start 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0c016e4700 1 -- start start 2024-03-18T15:51:12.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0c016e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0c016e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0bfc10c2c0 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0bfaffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0bfaffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60720/0 (socket says 172.21.15.150:60720) 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.169+0000 7f0bfaffd700 1 -- 172.21.15.150:0/1743793884 learned_addr learned my addr 172.21.15.150:0/1743793884 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0bfaffd700 1 -- 172.21.15.150:0/1743793884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bfc19daf0 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0bfaffd700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0bec005850 tx=0x7f0bec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bec026070 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bec00bbd0 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bfc19dde0 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bec003970 con 0x7f0bfc101ef0 2024-03-18T15:51:12.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.170+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bfc197850 con 0x7f0bfc101ef0 2024-03-18T15:51:12.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.171+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f0bec029070 con 0x7f0bfc101ef0 2024-03-18T15:51:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.171+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bfc1033d0 con 0x7f0bfc101ef0 2024-03-18T15:51:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.171+0000 7f0be3fff700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 0x7f0be403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.171+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0bec01e070 con 0x7f0bfc101ef0 2024-03-18T15:51:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.171+0000 7f0bfa7fc700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 0x7f0be403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.172+0000 7f0bfa7fc700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 0x7f0be403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0006fd0 tx=0x7f0bf0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:12.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.176+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bec013070 con 0x7f0bfc101ef0 2024-03-18T15:51:12.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:12.375+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) -- 0x7f0bfc198bb0 con 0x7f0be4038fc0 2024-03-18T15:51:12.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:12 smithi150 ceph-mon[31946]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:14.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:13 smithi150 ceph-mon[31946]: Deploying cephadm binary to smithi156 2024-03-18T15:51:16.749 INFO:teuthology.orchestra.run.smithi150.stdout:Added host 'smithi156' with addr '172.21.15.156' 2024-03-18T15:51:16.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.747+0000 7f0be3fff700 1 -- 172.21.15.150:0/1743793884 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0bfc198bb0 con 0x7f0be4038fc0 2024-03-18T15:51:16.749 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:16 smithi150 ceph-mon[31946]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:16.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 msgr2=0x7f0be403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:16.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 0x7f0be403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0006fd0 tx=0x7f0bf0006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:16.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 msgr2=0x7f0bfc19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:16.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0bec005850 tx=0x7f0bec005030 comp rx=0 tx=0).stop 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 shutdown_connections 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0be4038fc0 0x7f0be403b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 --2- 172.21.15.150:0/1743793884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bfc101ef0 0x7f0bfc19d5b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.749+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 >> 172.21.15.150:0/1743793884 conn(0x7f0bfc0fd7a0 msgr2=0x7f0bfc104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.750+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 shutdown_connections 2024-03-18T15:51:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:16.750+0000 7f0c016e4700 1 -- 172.21.15.150:0/1743793884 wait complete. 2024-03-18T15:51:17.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:51:17.617 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:18.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:18.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:17 smithi150 ceph-mon[31946]: Added host smithi156 2024-03-18T15:51:18.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:18.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:18.067 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:17 smithi150 ceph-mon[31946]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:18.979 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 -- 172.21.15.150:0/2633009111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 msgr2=0x7f96241011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 --2- 172.21.15.150:0/2633009111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f96241011f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9614009910 tx=0x7f9614009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 -- 172.21.15.150:0/2633009111 shutdown_connections 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 --2- 172.21.15.150:0/2633009111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f96241011f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 -- 172.21.15.150:0/2633009111 >> 172.21.15.150:0/2633009111 conn(0x7f96240fc6e0 msgr2=0x7f96240feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 -- 172.21.15.150:0/2633009111 shutdown_connections 2024-03-18T15:51:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.187+0000 7f9629091700 1 -- 172.21.15.150:0/2633009111 wait complete. 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.188+0000 7f9629091700 1 Processor -- start 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.188+0000 7f9629091700 1 -- start start 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.188+0000 7f9629091700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.188+0000 7f9629091700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f962410b1e0 con 0x7f9624100e10 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.188+0000 7f9622d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9622d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53716/0 (socket says 172.21.15.150:53716) 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9622d9d700 1 -- 172.21.15.150:0/222392945 learned_addr learned my addr 172.21.15.150:0/222392945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9622d9d700 1 -- 172.21.15.150:0/222392945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962419c930 con 0x7f9624100e10 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9622d9d700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9614005850 tx=0x7f9614005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9614026070 con 0x7f9624100e10 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962419cc20 con 0x7f9624100e10 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96241965b0 con 0x7f9624100e10 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f961400bbd0 con 0x7f9624100e10 2024-03-18T15:51:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.189+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9614003970 con 0x7f9624100e10 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.190+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f9614029070 con 0x7f9624100e10 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.190+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9610005330 con 0x7f9624100e10 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.190+0000 7f960bfff700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 0x7f960c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.190+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f961401e070 con 0x7f9624100e10 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.190+0000 7f962259c700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 0x7f960c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.191+0000 7f962259c700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 0x7f960c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9618006fd0 tx=0x7f9618006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:19.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.195+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9614013070 con 0x7f9624100e10 2024-03-18T15:51:19.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.396+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9610000bc0 con 0x7f960c039020 2024-03-18T15:51:19.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.397+0000 7f960bfff700 1 -- 172.21.15.150:0/222392945 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f9610000bc0 con 0x7f960c039020 2024-03-18T15:51:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:19.400 INFO:teuthology.orchestra.run.smithi150.stdout:[{"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 msgr2=0x7f960c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 0x7f960c03b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9618006fd0 tx=0x7f9618006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 msgr2=0x7f962419c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9614005850 tx=0x7f9614005030 comp rx=0 tx=0).stop 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 shutdown_connections 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f960c039020 0x7f960c03b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 --2- 172.21.15.150:0/222392945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9624100e10 0x7f962419c3f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:19.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.399+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 >> 172.21.15.150:0/222392945 conn(0x7f96240fc6e0 msgr2=0x7f9624103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:19.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.400+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 shutdown_connections 2024-03-18T15:51:19.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:19.400+0000 7f9629091700 1 -- 172.21.15.150:0/222392945 wait complete. 2024-03-18T15:51:19.988 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-03-18T15:51:19.988 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:51:19.988 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:51:20.008 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:51:20.008 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:20.066 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-03-18T15:51:20.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch host add smithi191 2024-03-18T15:51:20.306 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:20.551 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:20.551 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:20 smithi150 ceph-mon[31946]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:20.551 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:20 smithi150 ceph-mon[31946]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 -- 172.21.15.150:0/2398024587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 msgr2=0x7f19641011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 --2- 172.21.15.150:0/2398024587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f19641011d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1954009910 tx=0x7f1954009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 -- 172.21.15.150:0/2398024587 shutdown_connections 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 --2- 172.21.15.150:0/2398024587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f19641011d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 -- 172.21.15.150:0/2398024587 >> 172.21.15.150:0/2398024587 conn(0x7f19640fc6c0 msgr2=0x7f19640feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 -- 172.21.15.150:0/2398024587 shutdown_connections 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 -- 172.21.15.150:0/2398024587 wait complete. 2024-03-18T15:51:21.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.753+0000 7f1969f65700 1 Processor -- start 2024-03-18T15:51:21.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f1969f65700 1 -- start start 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f1969f65700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f1969f65700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f196410b1c0 con 0x7f1964100df0 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f19637fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f19637fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53732/0 (socket says 172.21.15.150:53732) 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f19637fe700 1 -- 172.21.15.150:0/3060015005 learned_addr learned my addr 172.21.15.150:0/3060015005 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:21.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f19637fe700 1 -- 172.21.15.150:0/3060015005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196419d950 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.754+0000 7f19637fe700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1954005850 tx=0x7f1954005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1954026070 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196419dc40 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195400bbd0 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1954003970 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19641975d0 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.755+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f1954029070 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.756+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19641022d0 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.756+0000 7f1960ff9700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 0x7f194c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.756+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f195401e070 con 0x7f1964100df0 2024-03-18T15:51:21.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.756+0000 7f1962ffd700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 0x7f194c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:21.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.756+0000 7f1962ffd700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 0x7f194c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1958006fd0 tx=0x7f1958006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:21.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.761+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1954013070 con 0x7f1964100df0 2024-03-18T15:51:21.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:21.960+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}) -- 0x7f1964198b00 con 0x7f194c038fc0 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:22.681 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:23.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:23 smithi150 ceph-mon[31946]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:23.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:23 smithi150 ceph-mon[31946]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T15:51:23.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:23 smithi150 ceph-mon[31946]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:24 smithi150 ceph-mon[31946]: Deploying cephadm binary to smithi191 2024-03-18T15:51:24.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:24 smithi150 ceph-mon[31946]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:24.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:24 smithi150 ceph-mon[31946]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:51:24.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:24 smithi150 ceph-mon[31946]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:26.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.118+0000 7f1960ff9700 1 -- 172.21.15.150:0/3060015005 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1964198b00 con 0x7f194c038fc0 2024-03-18T15:51:26.120 INFO:teuthology.orchestra.run.smithi150.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-03-18T15:51:26.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 msgr2=0x7f194c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:26.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 0x7f194c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1958006fd0 tx=0x7f1958006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 msgr2=0x7f196419d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1954005850 tx=0x7f1954005030 comp rx=0 tx=0).stop 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 shutdown_connections 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f194c038fc0 0x7f194c03b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 --2- 172.21.15.150:0/3060015005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1964100df0 0x7f196419d410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 >> 172.21.15.150:0/3060015005 conn(0x7f19640fc6c0 msgr2=0x7f1964069f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 shutdown_connections 2024-03-18T15:51:26.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:26.121+0000 7f1969f65700 1 -- 172.21.15.150:0/3060015005 wait complete. 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:26.438 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:26.716 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:51:26.948 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:27.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:27 smithi150 ceph-mon[31946]: Added host smithi191 2024-03-18T15:51:27.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:27.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:28.390 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:28 smithi150 ceph-mon[31946]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:28.390 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 -- 172.21.15.150:0/3485569634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 msgr2=0x7f27e81022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 --2- 172.21.15.150:0/3485569634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e81022b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009910 tx=0x7f27e4009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 -- 172.21.15.150:0/3485569634 shutdown_connections 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 --2- 172.21.15.150:0/3485569634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e81022b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 -- 172.21.15.150:0/3485569634 >> 172.21.15.150:0/3485569634 conn(0x7f27e80fd780 msgr2=0x7f27e80ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 -- 172.21.15.150:0/3485569634 shutdown_connections 2024-03-18T15:51:28.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.416+0000 7f27f0522700 1 -- 172.21.15.150:0/3485569634 wait complete. 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.417+0000 7f27f0522700 1 Processor -- start 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.417+0000 7f27f0522700 1 -- start start 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.417+0000 7f27f0522700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.417+0000 7f27f0522700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27e810c2a0 con 0x7f27e8101ed0 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27ee2be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:28.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27ee2be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53800/0 (socket says 172.21.15.150:53800) 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27ee2be700 1 -- 172.21.15.150:0/4290058599 learned_addr learned my addr 172.21.15.150:0/4290058599 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27ee2be700 1 -- 172.21.15.150:0/4290058599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e819d960 con 0x7f27e8101ed0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27ee2be700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005850 tx=0x7f27e4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e4026070 con 0x7f27e8101ed0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e400bbd0 con 0x7f27e8101ed0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e819dc50 con 0x7f27e8101ed0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e4003970 con 0x7f27e8101ed0 2024-03-18T15:51:28.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.418+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e81975e0 con 0x7f27e8101ed0 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.419+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f27e4029070 con 0x7f27e8101ed0 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.419+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e81033b0 con 0x7f27e8101ed0 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.419+0000 7f27df7fe700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 0x7f27d403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.419+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f27e401e070 con 0x7f27e8101ed0 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.419+0000 7f27edabd700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 0x7f27d403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:28.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.420+0000 7f27edabd700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 0x7f27d403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006fd0 tx=0x7f27d8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:28.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.424+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27e4013070 con 0x7f27e8101ed0 2024-03-18T15:51:28.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.623+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f27e8198b10 con 0x7f27d4039020 2024-03-18T15:51:28.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.624+0000 7f27df7fe700 1 -- 172.21.15.150:0/4290058599 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f27e8198b10 con 0x7f27d4039020 2024-03-18T15:51:28.625 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:51:28.625 INFO:teuthology.orchestra.run.smithi150.stdout:[{"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-03-18T15:51:28.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.626+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 msgr2=0x7f27d403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.626+0000 7f27f0522700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 0x7f27d403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006fd0 tx=0x7f27d8006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.626+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 msgr2=0x7f27e819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.626+0000 7f27f0522700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005850 tx=0x7f27e4005030 comp rx=0 tx=0).stop 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 shutdown_connections 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27d4039020 0x7f27d403b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 --2- 172.21.15.150:0/4290058599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27e8101ed0 0x7f27e819d420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 >> 172.21.15.150:0/4290058599 conn(0x7f27e80fd780 msgr2=0x7f27e81038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 shutdown_connections 2024-03-18T15:51:28.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:28.627+0000 7f27f0522700 1 -- 172.21.15.150:0/4290058599 wait complete. 2024-03-18T15:51:29.264 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T15:51:29.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T15:51:29.499 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:30.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:30 smithi150 ceph-mon[31946]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:51:30.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:30 smithi150 ceph-mon[31946]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 -- 172.21.15.150:0/2931078605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 msgr2=0x7fef201022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 --2- 172.21.15.150:0/2931078605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef201022e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009910 tx=0x7fef1c009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 -- 172.21.15.150:0/2931078605 shutdown_connections 2024-03-18T15:51:30.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 --2- 172.21.15.150:0/2931078605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef201022e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 -- 172.21.15.150:0/2931078605 >> 172.21.15.150:0/2931078605 conn(0x7fef200fd7b0 msgr2=0x7fef200ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 -- 172.21.15.150:0/2931078605 shutdown_connections 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.849+0000 7fef28596700 1 -- 172.21.15.150:0/2931078605 wait complete. 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef28596700 1 Processor -- start 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef28596700 1 -- start start 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef28596700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef28596700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fef2010c2d0 con 0x7fef20101f00 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef26332700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef26332700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53820/0 (socket says 172.21.15.150:53820) 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.850+0000 7fef26332700 1 -- 172.21.15.150:0/951716571 learned_addr learned my addr 172.21.15.150:0/951716571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef26332700 1 -- 172.21.15.150:0/951716571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2019d990 con 0x7fef20101f00 2024-03-18T15:51:30.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef26332700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef1c005850 tx=0x7fef1c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:30.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef1c026070 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef1c00bbd0 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2019dc80 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef1c003970 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.851+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef20197610 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.852+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fef1c029070 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.852+0000 7fef177fe700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 0x7fef0c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.852+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef201033e0 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.852+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fef1c01e070 con 0x7fef20101f00 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.852+0000 7fef25b31700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 0x7fef0c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:30.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.853+0000 7fef25b31700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 0x7fef0c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:30.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:30.861+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef1c030020 con 0x7fef20101f00 2024-03-18T15:51:31.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.071+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fef20063900 con 0x7fef20101f00 2024-03-18T15:51:31.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.423+0000 7fef177fe700 1 -- 172.21.15.150:0/951716571 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fef1c013070 con 0x7fef20101f00 2024-03-18T15:51:31.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.425+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 msgr2=0x7fef0c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:31.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.425+0000 7fef28596700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 0x7fef0c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 msgr2=0x7fef2019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef1c005850 tx=0x7fef1c005030 comp rx=0 tx=0).stop 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 shutdown_connections 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fef0c039020 0x7fef0c03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 --2- 172.21.15.150:0/951716571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef20101f00 0x7fef2019d450 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 >> 172.21.15.150:0/951716571 conn(0x7fef200fd7b0 msgr2=0x7fef201038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 shutdown_connections 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:31.426+0000 7fef28596700 1 -- 172.21.15.150:0/951716571 wait complete. 2024-03-18T15:51:31.427 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2024-03-18T15:51:31.756 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/951716571' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:51:31.756 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:31.756 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:31.756 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:31.756 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:31.757 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:51:31.757 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:31.757 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:32.075 INFO:tasks.cephadm:Adding mon.a on smithi150 2024-03-18T15:51:32.075 INFO:tasks.cephadm:Adding mon.b on smithi156 2024-03-18T15:51:32.076 INFO:tasks.cephadm:Adding mon.c on smithi191 2024-03-18T15:51:32.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c' 2024-03-18T15:51:32.303 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:32.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T15:51:32.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:32.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:32.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/951716571' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:51:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 -- 172.21.15.191:0/2119762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 msgr2=0x7fbc90103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 --2- 172.21.15.191:0/2119762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90103250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009910 tx=0x7fbc8c009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 -- 172.21.15.191:0/2119762452 shutdown_connections 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 --2- 172.21.15.191:0/2119762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90103250 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 -- 172.21.15.191:0/2119762452 >> 172.21.15.191:0/2119762452 conn(0x7fbc900fc6a0 msgr2=0x7fbc900feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 -- 172.21.15.191:0/2119762452 shutdown_connections 2024-03-18T15:51:33.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.776+0000 7fbc986e4700 1 -- 172.21.15.191:0/2119762452 wait complete. 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc986e4700 1 Processor -- start 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc986e4700 1 -- start start 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc986e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc986e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010c360 con 0x7fbc90102e70 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc96480700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc96480700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.191:58156/0 (socket says 172.21.15.191:58156) 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.777+0000 7fbc96480700 1 -- 172.21.15.191:0/1988846544 learned_addr learned my addr 172.21.15.191:0/1988846544 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc96480700 1 -- 172.21.15.191:0/1988846544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc901029c0 con 0x7fbc90102e70 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc96480700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006010 tx=0x7fbc8c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc8c026070 con 0x7fbc90102e70 2024-03-18T15:51:33.780 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c00bc30 con 0x7fbc90102e70 2024-03-18T15:51:33.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc90101010 con 0x7fbc90102e70 2024-03-18T15:51:33.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc8c003ab0 con 0x7fbc90102e70 2024-03-18T15:51:33.781 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.778+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc901014d0 con 0x7fbc90102e70 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.779+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9006a9a0 con 0x7fbc90102e70 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.779+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fbc8c029070 con 0x7fbc90102e70 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.779+0000 7fbc877fe700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 0x7fbc7c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.779+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbc8c01e070 con 0x7fbc90102e70 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.779+0000 7fbc95c7f700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 0x7fbc7c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:33.782 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.780+0000 7fbc95c7f700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 0x7fbc7c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:33.785 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.784+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc8c013070 con 0x7fbc90102e70 2024-03-18T15:51:33.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:33 smithi150 ceph-mon[31946]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:33.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:33 smithi150 ceph-mon[31946]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:51:33.992 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.991+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}) -- 0x7fbc90102070 con 0x7fbc7c038fc0 2024-03-18T15:51:33.998 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:33.998+0000 7fbc877fe700 1 -- 172.21.15.191:0/1988846544 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbc90102070 con 0x7fbc7c038fc0 2024-03-18T15:51:33.998 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-03-18T15:51:34.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 msgr2=0x7fbc7c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 0x7fbc7c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc80006fd0 tx=0x7fbc80006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 msgr2=0x7fbc90100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006010 tx=0x7fbc8c005050 comp rx=0 tx=0).stop 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 shutdown_connections 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbc7c038fc0 0x7fbc7c03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 --2- 172.21.15.191:0/1988846544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc90102e70 0x7fbc90100ad0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 >> 172.21.15.191:0/1988846544 conn(0x7fbc900fc6a0 msgr2=0x7fbc90103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 shutdown_connections 2024-03-18T15:51:34.001 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:34.000+0000 7fbc986e4700 1 -- 172.21.15.191:0/1988846544 wait complete. 2024-03-18T15:51:34.637 DEBUG:teuthology.orchestra.run.smithi156:mon.b> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T15:51:34.640 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T15:51:34.642 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:51:34.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:51:34.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:34.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:34.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:35.723 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:35.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:35 smithi150 ceph-mon[31946]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:35.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:35 smithi150 ceph-mon[31946]: Saving service mon spec with placement smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c;count:3 2024-03-18T15:51:35.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:35 smithi150 ceph-mon[31946]: Deploying daemon mon.c on smithi191 2024-03-18T15:51:36.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:36 smithi150 ceph-mon[31946]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:38.266 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- 172.21.15.191:0/2999176359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 msgr2=0x7f5b0c107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 --2- 172.21.15.191:0/2999176359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c107080 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009910 tx=0x7f5b00009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- 172.21.15.191:0/2999176359 shutdown_connections 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 --2- 172.21.15.191:0/2999176359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c107080 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- 172.21.15.191:0/2999176359 >> 172.21.15.191:0/2999176359 conn(0x7f5b0c0fbe10 msgr2=0x7f5b0c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- 172.21.15.191:0/2999176359 shutdown_connections 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- 172.21.15.191:0/2999176359 wait complete. 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 Processor -- start 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.264+0000 7f5b12adf700 1 -- start start 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b12adf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b12adf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10c420 con 0x7f5b0c106ca0 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b1087b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:38.267 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b1087b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.191:53528/0 (socket says 172.21.15.191:53528) 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b1087b700 1 -- 172.21.15.191:0/1611177748 learned_addr learned my addr 172.21.15.191:0/1611177748 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b1087b700 1 -- 172.21.15.191:0/1611177748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0c19d9a0 con 0x7f5b0c106ca0 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.265+0000 7f5b1087b700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005850 tx=0x7f5b00005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.266+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00026070 con 0x7f5b0c106ca0 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.266+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0c19dc30 con 0x7f5b0c106ca0 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.266+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0000bc30 con 0x7f5b0c106ca0 2024-03-18T15:51:38.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.266+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00003a20 con 0x7f5b0c106ca0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.266+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0c197620 con 0x7f5b0c106ca0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.267+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b0c1034f0 con 0x7f5b0c106ca0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.267+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f5b00029070 con 0x7f5b0c106ca0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.268+0000 7f5b09ffb700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 0x7f5afc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.268+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5b0001e070 con 0x7f5b0c106ca0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.268+0000 7f5b0bfff700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 0x7f5afc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.268+0000 7f5b0bfff700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 0x7f5afc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006fd0 tx=0x7f5af8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:38.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.273+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b00013070 con 0x7f5b0c106ca0 2024-03-18T15:51:38.562 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.562+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5b0c063900 con 0x7f5b0c106ca0 2024-03-18T15:51:38.563 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.562+0000 7f5b09ffb700 1 -- 172.21.15.191:0/1611177748 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f5b00030070 con 0x7f5b0c106ca0 2024-03-18T15:51:38.563 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:51:38.563 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","modified":"2024-03-18T15:49:56.417770Z","created":"2024-03-18T15:49:56.417770Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T15:51:38.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.564+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 msgr2=0x7f5afc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:38.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.564+0000 7f5b12adf700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 0x7f5afc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006fd0 tx=0x7f5af8006e40 comp rx=0 tx=0).stop 2024-03-18T15:51:38.565 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.564+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 msgr2=0x7f5b0c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.564+0000 7f5b12adf700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005850 tx=0x7f5b00005030 comp rx=0 tx=0).stop 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 shutdown_connections 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5afc038fc0 0x7f5afc03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 --2- 172.21.15.191:0/1611177748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0c106ca0 0x7f5b0c19d460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 >> 172.21.15.191:0/1611177748 conn(0x7f5b0c0fbe10 msgr2=0x7f5b0c109740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 shutdown_connections 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:38.565+0000 7f5b12adf700 1 -- 172.21.15.191:0/1611177748 wait complete. 2024-03-18T15:51:38.566 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-03-18T15:51:38.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:38 smithi150 ceph-mon[31946]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:39.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:38 smithi191 systemd[1]: Started Ceph mon.c for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:51:40.290 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:51:40.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:51:40.527 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: mon.a calling monitor election 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: mon.c calling monitor election 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: monmap epoch 2 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: last_changed 2024-03-18T15:51:38.891185+0000 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: min_mon_release 19 (squid) 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: election_strategy: 1 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: fsmap 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: mgrmap e13: x(active, since 48s) 2024-03-18T15:51:44.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: overall HEALTH_OK 2024-03-18T15:51:44.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:44.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:44.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:44.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: mon.a calling monitor election 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: mon.c calling monitor election 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: monmap epoch 2 2024-03-18T15:51:44.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: last_changed 2024-03-18T15:51:38.891185+0000 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: min_mon_release 19 (squid) 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: election_strategy: 1 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: fsmap 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: mgrmap e13: x(active, since 48s) 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: overall HEALTH_OK 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:44.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:45.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.107+0000 7f59e3984700 1 -- 172.21.15.191:0/1358651716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80035c0 msgr2=0x7f59c8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:45.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.107+0000 7f59e3984700 1 --2- 172.21.15.191:0/1358651716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80035c0 0x7f59c8005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59dc101ce0 tx=0x7f59d800feb0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.107+0000 7f59e3984700 1 -- 172.21.15.191:0/1358651716 shutdown_connections 2024-03-18T15:51:45.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.107+0000 7f59e3984700 1 --2- 172.21.15.191:0/1358651716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80035c0 0x7f59c8005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.107+0000 7f59e3984700 1 -- 172.21.15.191:0/1358651716 >> 172.21.15.191:0/1358651716 conn(0x7f59dc0fd8a0 msgr2=0x7f59dc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:45.109 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:44 smithi191 ceph-mon[36784]: Deploying daemon mon.b on smithi156 2024-03-18T15:51:45.109 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.108+0000 7f59e3984700 1 -- 172.21.15.191:0/1358651716 shutdown_connections 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.108+0000 7f59e3984700 1 -- 172.21.15.191:0/1358651716 wait complete. 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.108+0000 7f59e3984700 1 Processor -- start 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e3984700 1 -- start start 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e3984700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:45.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e3984700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 0x7f59dc1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e3984700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc109e40 con 0x7f59dc19dbe0 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e3984700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc109cc0 con 0x7f59c80035c0 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50498/0 (socket says 172.21.15.191:50498) 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 -- 172.21.15.191:0/2330841927 learned_addr learned my addr 172.21.15.191:0/2330841927 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e0f1f700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 0x7f59dc1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 -- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 msgr2=0x7f59dc19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 -- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 msgr2=0x7f59dc19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e1720700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e0f1f700 1 -- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 msgr2=0x7f59dc19d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e0f1f700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.109+0000 7f59e0f1f700 1 -- 172.21.15.191:0/2330841927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc19e160 con 0x7f59dc19dbe0 2024-03-18T15:51:45.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59e0f1f700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 0x7f59dc1977c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00b4d0 tx=0x7f59cc00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f59cc011630 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59cc011cb0 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc197d60 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f59cc010e60 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.110+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1982f0 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.111+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f59cc018020 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.111+0000 7f59d27fc700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 0x7f59c003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.111+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f59cc04ba60 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.111+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc1034f0 con 0x7f59dc19dbe0 2024-03-18T15:51:45.114 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.112+0000 7f59e1720700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 0x7f59c003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:45.115 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.112+0000 7f59e1720700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 0x7f59c003b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59d800fa60 tx=0x7f59d8005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:45.118 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.117+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59cc01f020 con 0x7f59dc19dbe0 2024-03-18T15:51:45.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:44 smithi150 ceph-mon[31946]: Deploying daemon mon.b on smithi156 2024-03-18T15:51:45.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:45.400 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.399+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f59dc198660 con 0x7f59dc19dbe0 2024-03-18T15:51:45.401 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.400+0000 7f59d27fc700 1 -- 172.21.15.191:0/2330841927 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f59cc018570 con 0x7f59dc19dbe0 2024-03-18T15:51:45.401 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:51:45.401 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","modified":"2024-03-18T15:51:38.891185Z","created":"2024-03-18T15:49:56.417770Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 msgr2=0x7f59c003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 0x7f59c003b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59d800fa60 tx=0x7f59d8005a90 comp rx=0 tx=0).stop 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 msgr2=0x7f59dc1977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 0x7f59dc1977c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00b4d0 tx=0x7f59cc00b890 comp rx=0 tx=0).stop 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 shutdown_connections 2024-03-18T15:51:45.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59c0038d60 0x7f59c003b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c80035c0 0x7f59dc19d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 --2- 172.21.15.191:0/2330841927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59dc19dbe0 0x7f59dc1977c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 >> 172.21.15.191:0/2330841927 conn(0x7f59dc0fd8a0 msgr2=0x7f59dc0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 shutdown_connections 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:45.403+0000 7f59e3984700 1 -- 172.21.15.191:0/2330841927 wait complete. 2024-03-18T15:51:45.405 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-03-18T15:51:46.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:45 smithi150 ceph-mon[31946]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:46.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:45 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/2330841927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:46.461 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:45 smithi191 ceph-mon[36784]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:46.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:45 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/2330841927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:46.969 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:51:46.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:51:47.212 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:51:47.728 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 podman[35899]: 2024-03-18 15:51:47.704359587 +0000 UTC m=+0.340294733 container start e907da403e8b1b6854077aa739a28a0ed42b383ad72f3fb4a0b7ad77905b1fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T15:51:47.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:51:47.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:51:47.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pidfile_write: ignore empty --pid-file 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: load: jerasure load: lrc 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Git sha 0 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: DB SUMMARY 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: DB Session ID: V34O86HARX6F6PY01CWI 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-18T15:51:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.env: 0x5558345d1c60 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.info_log: 0x555835c99ac0 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.statistics: (nil) 2024-03-18T15:51:48.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.use_fsync: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.db_log_dir: 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.wal_dir: 2024-03-18T15:51:48.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.write_buffer_manager: 0x555835c6c1e0 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:51:48.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.unordered_write: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.row_cache: None 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.wal_filter: None 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:51:48.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.wal_compression: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:51:48.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:51:48.025 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:51:48.025 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:51:48.025 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:51:48.026 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_open_files: -1 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:51:48.027 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Compression algorithms supported: 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kZSTD supported: 0 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.merge_operator: 2024-03-18T15:51:48.028 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_filter: None 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555835c99bc0) 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_type: 0 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_index_type: 0 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_shortening: 1 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:51:48.029 INFO:journalctl@ceph.mon.b.smithi156.stdout: checksum: 4 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: no_block_cache: 0 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache: 0x555835c9f090 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_options: 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: capacity : 536870912 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_shard_bits : 4 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: strict_capacity_limit : 0 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_compressed: (nil) 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: persistent_cache: (nil) 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size: 4096 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size_deviation: 10 2024-03-18T15:51:48.030 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_restart_interval: 16 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_block_restart_interval: 1 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: metadata_block_size: 4096 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: partition_filters: 0 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_delta_encoding: 1 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: filter_policy: bloomfilter 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: whole_key_filtering: 1 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: verify_compression: 0 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: format_version: 5 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_index_compression: 1 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_align: 0 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: max_auto_readahead_size: 262144 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: prepopulate_block_cache: 0 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:51:48.031 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression: NoCompression 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.num_levels: 7 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:51:48.032 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:51:48.033 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:51:48.034 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:51:48.035 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:51:48.036 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.ttl: 2592000 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:51:48.037 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cc3ab4dd-8e96-48a3-a86d-c0a1d48cee09 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777107734302, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777107734739, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710777107, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cc3ab4dd-8e96-48a3-a86d-c0a1d48cee09", "db_session_id": "V34O86HARX6F6PY01CWI", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777107734820, "job": 1, "event": "recovery_finished"} 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555835d92000 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: DB pointer 0x555835d7c000 2024-03-18T15:51:48.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** DB Stats ** 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:51:48.039 INFO:journalctl@ceph.mon.b.smithi156.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:51:48.040 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache BinnedLRUCache@0x555835c9f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: starting mon.b rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(???) e0 preinit fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:48.041 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).mds e1 new map 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: e1 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: legacy client fscid: -1 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout: No filesystems configured 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T15:51:48.042 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T15:51:48.043 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Deploying cephadm binary to smithi191 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Added host smithi191 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/951716571' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.045 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/951716571' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.046 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Saving service mon spec with placement smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c;count:3 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Deploying daemon mon.c on smithi191 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.a calling monitor election 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.c calling monitor election 2024-03-18T15:51:48.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: monmap epoch 2 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: last_changed 2024-03-18T15:51:38.891185+0000 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: min_mon_release 19 (squid) 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: election_strategy: 1 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:48.048 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: fsmap 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mgrmap e13: x(active, since 48s) 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: overall HEALTH_OK 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: Deploying daemon mon.b on smithi156 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/2330841927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 ceph-mon[35919]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T15:51:48.049 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 bash[35899]: e907da403e8b1b6854077aa739a28a0ed42b383ad72f3fb4a0b7ad77905b1fab 2024-03-18T15:51:48.050 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:47 smithi156 systemd[1]: Started Ceph mon.b for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:51:52.990 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 -- 172.21.15.191:0/784020191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14080035c0 msgr2=0x7f1408005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 --2- 172.21.15.191:0/784020191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14080035c0 0x7f1408005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f141c06a4f0 tx=0x7f141800fe60 comp rx=0 tx=0).stop 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 -- 172.21.15.191:0/784020191 shutdown_connections 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 --2- 172.21.15.191:0/784020191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14080035c0 0x7f1408005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 -- 172.21.15.191:0/784020191 >> 172.21.15.191:0/784020191 conn(0x7f141c0fc7f0 msgr2=0x7f141c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 -- 172.21.15.191:0/784020191 shutdown_connections 2024-03-18T15:51:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.989+0000 7f1423389700 1 -- 172.21.15.191:0/784020191 wait complete. 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 Processor -- start 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 -- start start 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 0x7f141c1a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f141c109e30 con 0x7f141c1a3c90 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f141c109cb0 con 0x7f14080035c0 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1423389700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f141c109fb0 con 0x7f141c1011e0 2024-03-18T15:51:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1420924700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1420924700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:56916/0 (socket says 172.21.15.191:56916) 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1420924700 1 -- 172.21.15.191:0/3257208383 learned_addr learned my addr 172.21.15.191:0/3257208383 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1421125700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.991+0000 7f1421926700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 0x7f141c1a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1420924700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 msgr2=0x7f141c101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1420924700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 msgr2=0x7f141c101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421125700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 msgr2=0x7f141c100ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1420924700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421125700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 msgr2=0x7f141c100ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421125700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:51:52.994 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421125700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1420924700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 msgr2=0x7f141c101660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 msgr2=0x7f141c100ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 -- 172.21.15.191:0/3257208383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c1a67b0 con 0x7f141c1a3c90 2024-03-18T15:51:52.995 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1421926700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 0x7f141c1a6080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f141400bd50 tx=0x7f141400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:52.996 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414019070 con 0x7f141c1a3c90 2024-03-18T15:51:52.996 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.992+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c1a6aa0 con 0x7f141c1a3c90 2024-03-18T15:51:52.996 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.993+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f141400cd10 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.993+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14140127e0 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.993+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c1a7030 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.993+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f141401c070 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.994+0000 7f14127fc700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 0x7f140003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.994+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141c063900 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.994+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1414014ab0 con 0x7f141c1a3c90 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.994+0000 7f1421125700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 0x7f140003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:52.997 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.995+0000 7f1421125700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 0x7f140003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1418020040 tx=0x7f1418001e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:53.000 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:52.999+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1414013e80 con 0x7f141c1a3c90 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: mon.a calling monitor election 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: mon.c calling monitor election 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: mon.b calling monitor election 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: monmap epoch 3 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: last_changed 2024-03-18T15:51:47.757079+0000 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: min_mon_release 19 (squid) 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: election_strategy: 1 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: fsmap 2024-03-18T15:51:53.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: mgrmap e13: x(active, since 57s) 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: overall HEALTH_OK 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: mon.a calling monitor election 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: mon.c calling monitor election 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: mon.b calling monitor election 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: monmap epoch 3 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: last_changed 2024-03-18T15:51:47.757079+0000 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: min_mon_release 19 (squid) 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: election_strategy: 1 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:53.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: fsmap 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: mgrmap e13: x(active, since 57s) 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: overall HEALTH_OK 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.291 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.290+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f141c1021d0 con 0x7f141c1a3c90 2024-03-18T15:51:53.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.291+0000 7f14127fc700 1 -- 172.21.15.191:0/3257208383 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f141c1021d0 con 0x7f141c1a3c90 2024-03-18T15:51:53.292 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-18T15:51:53.292 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","modified":"2024-03-18T15:51:47.757079Z","created":"2024-03-18T15:49:56.417770Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 msgr2=0x7f140003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 0x7f140003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1418020040 tx=0x7f1418001e70 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 msgr2=0x7f141c1a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 0x7f141c1a6080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f141400bd50 tx=0x7f141400be30 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 shutdown_connections 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c1a3c90 0x7f141c1a6080 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.293+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1400039450 0x7f140003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.294+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14080035c0 0x7f141c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.294+0000 7f1423389700 1 --2- 172.21.15.191:0/3257208383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f141c1011e0 0x7f141c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.294+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 >> 172.21.15.191:0/3257208383 conn(0x7f141c0fc7f0 msgr2=0x7f141c103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.294+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 shutdown_connections 2024-03-18T15:51:53.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:53.294+0000 7f1423389700 1 -- 172.21.15.191:0/3257208383 wait complete. 2024-03-18T15:51:53.296 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: mon.a calling monitor election 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: mon.c calling monitor election 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: mon.b calling monitor election 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: monmap epoch 3 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: last_changed 2024-03-18T15:51:47.757079+0000 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: created 2024-03-18T15:49:56.417770+0000 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: min_mon_release 19 (squid) 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: election_strategy: 1 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: 0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-18T15:51:53.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: 1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.c 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: fsmap 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: mgrmap e13: x(active, since 57s) 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: overall HEALTH_OK 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.344 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:53.815 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T15:51:53.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T15:51:54.068 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:51:54.178 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.178 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:53 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/3257208383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:53 smithi191 ceph-mon[36784]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:54.179 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:54.189 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.189 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:53 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/3257208383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:53 smithi150 ceph-mon[31946]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:54.190 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:54.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:53 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/3257208383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:51:54.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:53 smithi156 ceph-mon[35919]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:54.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.520 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:55.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 -- 172.21.15.150:0/1107111314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 msgr2=0x7f46301011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 --2- 172.21.15.150:0/1107111314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301011d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f462c009910 tx=0x7f462c009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 -- 172.21.15.150:0/1107111314 shutdown_connections 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 --2- 172.21.15.150:0/1107111314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301011d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 -- 172.21.15.150:0/1107111314 >> 172.21.15.150:0/1107111314 conn(0x7f46300fc6c0 msgr2=0x7f46300feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 -- 172.21.15.150:0/1107111314 shutdown_connections 2024-03-18T15:51:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.745+0000 7f46385f3700 1 -- 172.21.15.150:0/1107111314 wait complete. 2024-03-18T15:51:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 Processor -- start 2024-03-18T15:51:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 -- start start 2024-03-18T15:51:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4630198de0 0x7f4630199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 0x7f46301a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f463010b1c0 con 0x7f4630100df0 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f463010b040 con 0x7f46301a1910 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f46385f3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f463010b340 con 0x7f4630198de0 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f463638f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f463638f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56608/0 (socket says 172.21.15.150:56608) 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f463638f700 1 -- 172.21.15.150:0/1260797683 learned_addr learned my addr 172.21.15.150:0/1260797683 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f4636b90700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 0x7f46301a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.746+0000 7f4635b8e700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4630198de0 0x7f4630199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 -- 172.21.15.150:0/1260797683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4630198de0 msgr2=0x7f4630199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4630198de0 0x7f4630199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 -- 172.21.15.150:0/1260797683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 msgr2=0x7f46301a3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 0x7f46301a3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 -- 172.21.15.150:0/1260797683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46301a4510 con 0x7f4630100df0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f4636b90700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 0x7f46301a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f463638f700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f462c005850 tx=0x7f462c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c026070 con 0x7f4630100df0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46301a4800 con 0x7f4630100df0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f462c00b6f0 con 0x7f4630100df0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0038e0 con 0x7f4630100df0 2024-03-18T15:51:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.747+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46301a4d90 con 0x7f4630100df0 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.748+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f462c029070 con 0x7f4630100df0 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.748+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46301022d0 con 0x7f4630100df0 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.748+0000 7f46277fe700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 0x7f461c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.748+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f462c04cb90 con 0x7f4630100df0 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.749+0000 7f4635b8e700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 0x7f461c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.749+0000 7f4635b8e700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 0x7f461c03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4620009ad0 tx=0x7f4620009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:55.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.754+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f462c013070 con 0x7f4630100df0 2024-03-18T15:51:55.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.957+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f46301a4ff0 con 0x7f4630100df0 2024-03-18T15:51:55.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.958+0000 7f46277fe700 1 -- 172.21.15.150:0/1260797683 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f462c030020 con 0x7f4630100df0 2024-03-18T15:51:55.959 INFO:teuthology.orchestra.run.smithi150.stdout:# minimal ceph.conf for 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:55.959 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-03-18T15:51:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:51:55.959 INFO:teuthology.orchestra.run.smithi150.stdout: mon_host = [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 msgr2=0x7f461c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 0x7f461c03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4620009ad0 tx=0x7f4620009210 comp rx=0 tx=0).stop 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 msgr2=0x7f46301988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f462c005850 tx=0x7f462c004f80 comp rx=0 tx=0).stop 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 shutdown_connections 2024-03-18T15:51:55.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f461c039450 0x7f461c03b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46301a1910 0x7f46301a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4630100df0 0x7f46301988a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 --2- 172.21.15.150:0/1260797683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4630198de0 0x7f4630199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 >> 172.21.15.150:0/1260797683 conn(0x7f46300fc6c0 msgr2=0x7f46301027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 shutdown_connections 2024-03-18T15:51:55.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:51:55.960+0000 7f46385f3700 1 -- 172.21.15.150:0/1260797683 wait complete. 2024-03-18T15:51:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T15:51:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T15:51:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T15:51:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:56.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:56.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: Reconfiguring daemon mon.a on smithi150 2024-03-18T15:51:56.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:56.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:56 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1260797683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:56.501 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T15:51:56.501 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T15:51:56.501 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T15:51:56.501 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.501 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:56.502 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:56.503 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:56.503 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: Reconfiguring daemon mon.a on smithi150 2024-03-18T15:51:56.503 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:56.503 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.503 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:56 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1260797683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.508 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: Reconfiguring daemon mon.a on smithi150 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:56.509 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:56 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1260797683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:57.387 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T15:51:57.387 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:57.387 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:51:57.416 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:51:57.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:57.484 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:51:57.484 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:51:57.513 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:51:57.513 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:57.581 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:51:57.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:51:57.615 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:51:57.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:51:57.686 INFO:tasks.cephadm:Adding mgr.x on smithi150 2024-03-18T15:51:57.686 INFO:tasks.cephadm:Adding mgr.y on smithi156 2024-03-18T15:51:57.686 INFO:tasks.cephadm:Adding mgr.z on smithi191 2024-03-18T15:51:57.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi150=x;smithi156=y;smithi191=z' 2024-03-18T15:51:57.959 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:51:58.267 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:57 smithi156 ceph-mon[35919]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:58 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:58 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:58 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:58 smithi156 ceph-mon[35919]: Reconfiguring daemon mon.b on smithi156 2024-03-18T15:51:58.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:58 smithi156 ceph-mon[35919]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:58.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:58.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: Reconfiguring daemon mon.b on smithi156 2024-03-18T15:51:58.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:57 smithi191 ceph-mon[36784]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: Reconfiguring mon.b (monmap changed)... 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: Reconfiguring daemon mon.b on smithi156 2024-03-18T15:51:58.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:57 smithi150 ceph-mon[31946]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.465+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/1580724806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 msgr2=0x7fcf78104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.465+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/1580724806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78104cb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcf68009910 tx=0x7fcf68009c20 comp rx=0 tx=0).stop 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.465+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/1580724806 shutdown_connections 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.465+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/1580724806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78104cb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.465+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/1580724806 >> 172.21.15.191:0/1580724806 conn(0x7fcf780fc2d0 msgr2=0x7fcf780fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/1580724806 shutdown_connections 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/1580724806 wait complete. 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 Processor -- start 2024-03-18T15:51:59.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- start start 2024-03-18T15:51:59.470 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:59.470 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf78100870 0x7fcf78100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf78101230 0x7fcf781a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78107e60 con 0x7fcf781028c0 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78107ce0 con 0x7fcf78101230 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.466+0000 7fcf7d9b2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78107fe0 con 0x7fcf78100870 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.191:33498/0 (socket says 172.21.15.191:33498) 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 -- 172.21.15.191:0/3781368411 learned_addr learned my addr 172.21.15.191:0/3781368411 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf6e7fc700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf78100870 0x7fcf78100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf777fe700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf78101230 0x7fcf781a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 -- 172.21.15.191:0/3781368411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf78100870 msgr2=0x7fcf78100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf78100870 0x7fcf78100cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 -- 172.21.15.191:0/3781368411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf78101230 msgr2=0x7fcf781a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:59.471 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf78101230 0x7fcf781a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 -- 172.21.15.191:0/3781368411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf781a1bc0 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf76ffd700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf68005950 tx=0x7fcf68004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68026070 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6800b6f0 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68003a60 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf781a1eb0 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.467+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf781a2390 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.468+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf78063900 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.468+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fcf68029070 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.469+0000 7fcf7c9b0700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 0x7fcf6403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.469+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcf6804cc00 con 0x7fcf781028c0 2024-03-18T15:51:59.472 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.469+0000 7fcf6e7fc700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 0x7fcf6403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:51:59.473 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.469+0000 7fcf6e7fc700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 0x7fcf6403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009ad0 tx=0x7fcf60009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:51:59.476 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.474+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf68013070 con 0x7fcf781028c0 2024-03-18T15:51:59.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.688+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi150=x;smithi156=y;smithi191=z", "target": ["mon-mgr", ""]}) -- 0x7fcf780627a0 con 0x7fcf64039450 2024-03-18T15:51:59.696 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.694+0000 7fcf7c9b0700 1 -- 172.21.15.191:0/3781368411 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcf780627a0 con 0x7fcf64039450 2024-03-18T15:51:59.696 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-03-18T15:51:59.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:59.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:59.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:59.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:59.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:51:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 msgr2=0x7fcf6403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 0x7fcf6403b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009ad0 tx=0x7fcf60009210 comp rx=0 tx=0).stop 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 msgr2=0x7fcf78102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf68005950 tx=0x7fcf68004f80 comp rx=0 tx=0).stop 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 shutdown_connections 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcf64039450 0x7fcf6403b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.698 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf78101230 0x7fcf781a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.699 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf781028c0 0x7fcf78102220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.699 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 --2- 172.21.15.191:0/3781368411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf78100870 0x7fcf78100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:51:59.699 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 >> 172.21.15.191:0/3781368411 conn(0x7fcf780fc2d0 msgr2=0x7fcf780fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:51:59.699 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 shutdown_connections 2024-03-18T15:51:59.699 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:51:59.696+0000 7fcf7d9b2700 1 -- 172.21.15.191:0/3781368411 wait complete. 2024-03-18T15:51:59.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:59.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:51:59.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:51:59.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:51:59.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:51:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:52:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:52:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:51:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:00.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:00 smithi150 ceph-mon[31946]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:52:00.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:00 smithi150 ceph-mon[31946]: Reconfiguring daemon mon.c on smithi191 2024-03-18T15:52:00.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:00 smithi150 ceph-mon[31946]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:00.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:00.895 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:00 smithi191 ceph-mon[36784]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:52:00.895 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:00 smithi191 ceph-mon[36784]: Reconfiguring daemon mon.c on smithi191 2024-03-18T15:52:00.895 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:00 smithi191 ceph-mon[36784]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:00.895 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:00.922 DEBUG:teuthology.orchestra.run.smithi156:mgr.y> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T15:52:00.925 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T15:52:00.928 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T15:52:00.928 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:52:00.928 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:52:00.944 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:52:00.944 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-03-18T15:52:00.952 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:00 smithi156 ceph-mon[35919]: Reconfiguring mon.c (monmap changed)... 2024-03-18T15:52:00.952 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:00 smithi156 ceph-mon[35919]: Reconfiguring daemon mon.c on smithi191 2024-03-18T15:52:00.952 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:00 smithi156 ceph-mon[35919]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:00.952 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-18 15:51:05.095190679 +0000 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-18 15:45:58.696620265 +0000 2024-03-18T15:52:00.999 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-18 15:45:58.696620265 +0000 2024-03-18T15:52:01.000 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-18 15:45:58.696620265 +0000 2024-03-18T15:52:01.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:52:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-18T15:52:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-18T15:52:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000195074 s, 2.6 MB/s 2024-03-18T15:52:01.067 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:52:01.123 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-03-18T15:52:01.179 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:52:01.179 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-18 15:51:05.130189963 +0000 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-18 15:45:59.038613121 +0000 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-18 15:45:59.038613121 +0000 2024-03-18T15:52:01.180 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-18 15:45:59.038613121 +0000 2024-03-18T15:52:01.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:52:01.246 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-18T15:52:01.246 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-18T15:52:01.246 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000208511 s, 2.5 MB/s 2024-03-18T15:52:01.247 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:52:01.303 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-18 15:51:05.169189165 +0000 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-18 15:45:59.373606124 +0000 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-18 15:45:59.373606124 +0000 2024-03-18T15:52:01.358 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-18 15:45:59.373606124 +0000 2024-03-18T15:52:01.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:52:01.424 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-18T15:52:01.424 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-18T15:52:01.424 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000230554 s, 2.2 MB/s 2024-03-18T15:52:01.425 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:52:01.480 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-18 15:51:05.200188530 +0000 2024-03-18T15:52:01.535 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-18 15:45:59.727598729 +0000 2024-03-18T15:52:01.536 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-18 15:45:59.727598729 +0000 2024-03-18T15:52:01.536 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-18 15:45:59.727598729 +0000 2024-03-18T15:52:01.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:52:01.601 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-18T15:52:01.601 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-18T15:52:01.601 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000188684 s, 2.7 MB/s 2024-03-18T15:52:01.602 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:52:01.658 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-18T15:52:01.658 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:52:01.677 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:52:01.677 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-18 15:51:47.189143478 +0000 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-18 15:46:39.777985363 +0000 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-18 15:46:39.777985363 +0000 2024-03-18T15:52:01.735 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-18 15:46:39.777985363 +0000 2024-03-18T15:52:01.735 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:52:01.801 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi150=x;smithi156=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:01.801 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: Saving service mgr spec with placement smithi150=x;smithi156=y;smithi191=z;count:3 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:01.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:01 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.804 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-18T15:52:01.804 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-18T15:52:01.804 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000102079 s, 5.0 MB/s 2024-03-18T15:52:01.805 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:52:01.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi150=x;smithi156=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:01.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: Saving service mgr spec with placement smithi150=x;smithi156=y;smithi191=z;count:3 2024-03-18T15:52:01.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:01 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.863 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-03-18T15:52:01.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi150=x;smithi156=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:01.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: Saving service mgr spec with placement smithi150=x;smithi156=y;smithi191=z;count:3 2024-03-18T15:52:01.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:01.864 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:01 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-18 15:51:47.190143464 +0000 2024-03-18T15:52:01.919 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-18 15:46:40.126980536 +0000 2024-03-18T15:52:01.920 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-18 15:46:40.126980536 +0000 2024-03-18T15:52:01.920 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-18 15:46:40.126980536 +0000 2024-03-18T15:52:01.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:52:01.986 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-18T15:52:01.987 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-18T15:52:01.987 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.00012066 s, 4.2 MB/s 2024-03-18T15:52:01.988 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:52:02.044 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-03-18T15:52:02.100 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:52:02.100 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.100 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-18 15:51:47.190143464 +0000 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-18 15:46:40.455975987 +0000 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-18 15:46:40.455975987 +0000 2024-03-18T15:52:02.101 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-18 15:46:40.455975987 +0000 2024-03-18T15:52:02.101 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:52:02.166 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-18T15:52:02.166 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-18T15:52:02.166 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000116761 s, 4.4 MB/s 2024-03-18T15:52:02.167 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:52:02.223 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-18 15:51:47.190143464 +0000 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-18 15:46:40.780971492 +0000 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-18 15:46:40.780971492 +0000 2024-03-18T15:52:02.278 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-18 15:46:40.780971492 +0000 2024-03-18T15:52:02.279 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:52:02.344 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-18T15:52:02.344 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-18T15:52:02.344 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.4709e-05 s, 5.4 MB/s 2024-03-18T15:52:02.345 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:52:02.400 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-18T15:52:02.400 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T15:52:02.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T15:52:02.417 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-18 15:51:37.984381046 +0000 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-18 15:46:29.038911888 +0000 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-18 15:46:29.038911888 +0000 2024-03-18T15:52:02.473 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-18 15:46:29.038911888 +0000 2024-03-18T15:52:02.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T15:52:02.539 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-18T15:52:02.539 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-18T15:52:02.539 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000261947 s, 2.0 MB/s 2024-03-18T15:52:02.540 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T15:52:02.596 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-18 15:51:37.985381028 +0000 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-18 15:46:29.377905566 +0000 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-18 15:46:29.377905566 +0000 2024-03-18T15:52:02.651 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-18 15:46:29.377905566 +0000 2024-03-18T15:52:02.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T15:52:02.716 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-18T15:52:02.716 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-18T15:52:02.716 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000198708 s, 2.6 MB/s 2024-03-18T15:52:02.718 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T15:52:02.774 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-18T15:52:02.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:02 smithi191 ceph-mon[36784]: Deploying daemon mgr.z on smithi191 2024-03-18T15:52:02.789 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:02 smithi191 ceph-mon[36784]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.803 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-18 15:51:37.985381028 +0000 2024-03-18T15:52:02.804 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-18 15:46:29.708899393 +0000 2024-03-18T15:52:02.804 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-18 15:46:29.708899393 +0000 2024-03-18T15:52:02.804 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-18 15:46:29.708899393 +0000 2024-03-18T15:52:02.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T15:52:02.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:02 smithi150 ceph-mon[31946]: Deploying daemon mgr.z on smithi191 2024-03-18T15:52:02.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:02 smithi150 ceph-mon[31946]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:02.869 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-18T15:52:02.869 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-18T15:52:02.869 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00019273 s, 2.7 MB/s 2024-03-18T15:52:02.870 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T15:52:02.930 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T15:52:02.985 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-18 15:51:37.986381010 +0000 2024-03-18T15:52:02.986 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-18 15:46:30.029893407 +0000 2024-03-18T15:52:02.986 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-18 15:46:30.029893407 +0000 2024-03-18T15:52:02.986 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-18 15:46:30.029893407 +0000 2024-03-18T15:52:02.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T15:52:03.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:02 smithi156 ceph-mon[35919]: Deploying daemon mgr.z on smithi191 2024-03-18T15:52:03.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:02 smithi156 ceph-mon[35919]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:03.058 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-18T15:52:03.058 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-18T15:52:03.058 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000229013 s, 2.2 MB/s 2024-03-18T15:52:03.060 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T15:52:03.119 INFO:tasks.cephadm:Deploying osd.0 on smithi150 with /dev/vg_nvme/lv_4... 2024-03-18T15:52:03.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:52:03.361 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:52:03.459 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 systemd[1]: Starting Ceph mgr.z for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:52:03.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 podman[38623]: 2024-03-18 15:52:03.458740494 +0000 UTC m=+0.112024585 container create b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T15:52:03.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 podman[38623]: 2024-03-18 15:52:03.364391937 +0000 UTC m=+0.017675989 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:52:03.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 podman[38623]: 2024-03-18 15:52:03.655958103 +0000 UTC m=+0.309242156 container init b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:52:03.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 podman[38623]: 2024-03-18 15:52:03.658408762 +0000 UTC m=+0.311692814 container start b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T15:52:03.713 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 ceph-mgr[38642]: -- 172.21.15.191:0/4007210709 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7222265a0 con 0x55b7221fb000 2024-03-18T15:52:04.202 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 bash[38623]: b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f 2024-03-18T15:52:04.203 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:03 smithi191 systemd[1]: Started Ceph mgr.z for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:52:04.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:04.200+0000 7fbc65727200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:52:04.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:04.293+0000 7fbc65727200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:52:04.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:04.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.705 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:04.705 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:04.705 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:04.705 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:04.787 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:04.928 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:05.159 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:52:05.159 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:52:05.159 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-18T15:52:05.159 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-18T15:52:05.160 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122232 s, 858 MB/s 2024-03-18T15:52:05.160 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-18T15:52:05.212 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:04.925+0000 7fbc65727200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:52:05.212 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.017+0000 7fbc65727200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:52:05.212 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.157+0000 7fbc65727200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:52:05.541 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.251+0000 7fbc65727200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:52:05.541 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.333+0000 7fbc65727200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:52:05.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-03-18T15:52:05.767 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:05 smithi150 ceph-mon[31946]: Deploying daemon mgr.y on smithi156 2024-03-18T15:52:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:05 smithi156 ceph-mon[35919]: Deploying daemon mgr.y on smithi156 2024-03-18T15:52:05.768 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:05 smithi156 systemd[1]: Starting Ceph mgr.y for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:52:05.835 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-mon[36784]: Deploying daemon mgr.y on smithi156 2024-03-18T15:52:05.835 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.596+0000 7fbc65727200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:52:05.835 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.703+0000 7fbc65727200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:52:05.835 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.788+0000 7fbc65727200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:52:05.977 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:52:06.094 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:05.884+0000 7fbc65727200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:52:06.094 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:06.067+0000 7fbc65727200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:52:06.186 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:05 smithi156 podman[37286]: 2024-03-18 15:52:05.89238972 +0000 UTC m=+0.109463728 container create 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T15:52:06.187 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:05 smithi156 podman[37286]: 2024-03-18 15:52:05.800655146 +0000 UTC m=+0.017729170 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:52:06.187 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 podman[37286]: 2024-03-18 15:52:06.102883176 +0000 UTC m=+0.319957188 container init 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:52:06.187 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 podman[37286]: 2024-03-18 15:52:06.105404991 +0000 UTC m=+0.322479008 container start 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:52:06.187 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mgr[37305]: -- 172.21.15.156:0/4063884493 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e95d70e5a0 con 0x55e95d6e2c00 2024-03-18T15:52:06.187 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 bash[37286]: 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b 2024-03-18T15:52:06.517 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 systemd[1]: Started Ceph mgr.y for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:52:06.730 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:06.730 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.730 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.730 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.731 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:06.661+0000 7f97c4233200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:52:07.018 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:06.757+0000 7f97c4233200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:52:07.423 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:07.148+0000 7fbc65727200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:52:07.423 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:07.280+0000 7fbc65727200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:52:07.447 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:07.439+0000 7f97c4233200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:52:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 -- 172.21.15.150:0/1649474732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4102da0 msgr2=0x7f20d4109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:07.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 --2- 172.21.15.150:0/1649474732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4102da0 0x7f20d4109650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b0d0 tx=0x7f20cc00b3e0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 -- 172.21.15.150:0/1649474732 shutdown_connections 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 --2- 172.21.15.150:0/1649474732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4102da0 0x7f20d4109650 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 --2- 172.21.15.150:0/1649474732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4101200 0x7f20d4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 --2- 172.21.15.150:0/1649474732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102400 0x7f20d4102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 -- 172.21.15.150:0/1649474732 >> 172.21.15.150:0/1649474732 conn(0x7f20d40fc7a0 msgr2=0x7f20d40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 -- 172.21.15.150:0/1649474732 shutdown_connections 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 -- 172.21.15.150:0/1649474732 wait complete. 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.536+0000 7f20d9de7700 1 Processor -- start 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 -- start start 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4102400 0x7f20d41037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102da0 0x7f20d4103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:07.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f20d410ce30 con 0x7f20d4101200 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20d410ccb0 con 0x7f20d4102da0 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d9de7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f20d410cfb0 con 0x7f20d4102400 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38708/0 (socket says 172.21.15.150:38708) 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 -- 172.21.15.150:0/3371164419 learned_addr learned my addr 172.21.15.150:0/3371164419 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d3fff700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102da0 0x7f20d4103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20caffd700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4102400 0x7f20d41037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 -- 172.21.15.150:0/3371164419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4102400 msgr2=0x7f20d41037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4102400 0x7f20d41037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 -- 172.21.15.150:0/3371164419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102da0 msgr2=0x7f20d4103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102da0 0x7f20d4103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 -- 172.21.15.150:0/3371164419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d41045d0 con 0x7f20d4101200 2024-03-18T15:52:07.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.537+0000 7f20d37fe700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f20c400d980 tx=0x7f20c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4017070 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c400fd70 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d41a2790 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c400e3b0 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41a2ca0 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.538+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f20c400e550 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.539+0000 7f20d17fa700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 0x7f20c003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.539+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d4068f90 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.539+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f20c404bbd0 con 0x7f20d4101200 2024-03-18T15:52:07.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.539+0000 7f20caffd700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 0x7f20c003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:07.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.539+0000 7f20caffd700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 0x7f20c003b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20bc007be0 tx=0x7f20bc0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:07.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.544+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20c4010460 con 0x7f20d4101200 2024-03-18T15:52:07.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:07.622+0000 7fbc65727200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:52:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:07.749+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f20d41a2f00 con 0x7f20c0039200 2024-03-18T15:52:07.767 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:07.532+0000 7f97c4233200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:52:07.768 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:07.676+0000 7f97c4233200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:52:08.111 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:07.712+0000 7fbc65727200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:52:08.112 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:07.926+0000 7fbc65727200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:52:08.112 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:08.110+0000 7fbc65727200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:52:08.127 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:07.778+0000 7f97c4233200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:52:08.127 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:07.860+0000 7f97c4233200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:52:08.127 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:08.126+0000 7f97c4233200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:52:08.410 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:08.233+0000 7f97c4233200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:52:08.411 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:08.317+0000 7f97c4233200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:52:08.411 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:08.409+0000 7f97c4233200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:52:08.655 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:08.393+0000 7fbc65727200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:08.654+0000 7fbc65727200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.656 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:08.657 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.670 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.671 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:08.671 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.671 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:08.671 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:08.671 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:08.584+0000 7f97c4233200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T15:52:08.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:08.961 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:08.889+0000 7fbc65727200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:52:09.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:09 smithi150 ceph-mon[31946]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:09.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:09 smithi150 ceph-mon[31946]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:52:09.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:09 smithi150 ceph-mon[31946]: Reconfiguring daemon mgr.x on smithi150 2024-03-18T15:52:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:09 smithi191 ceph-mon[36784]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:09 smithi191 ceph-mon[36784]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:52:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:09 smithi191 ceph-mon[36784]: Reconfiguring daemon mgr.x on smithi150 2024-03-18T15:52:09.962 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:09.693+0000 7fbc65727200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:52:10.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:09 smithi156 ceph-mon[35919]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:10.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:09 smithi156 ceph-mon[35919]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T15:52:10.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:09 smithi156 ceph-mon[35919]: Reconfiguring daemon mgr.x on smithi150 2024-03-18T15:52:10.011 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:09.549+0000 7f97c4233200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:52:10.012 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:09.678+0000 7f97c4233200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:52:10.012 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:10.009+0000 7f97c4233200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:52:10.267 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:10.097+0000 7f97c4233200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:52:10.553 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:10.261+0000 7fbc65727200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:52:10.553 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:10.358+0000 7fbc65727200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:52:10.554 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:10.288+0000 7f97c4233200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:52:10.554 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:10.470+0000 7f97c4233200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:52:10.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:52:10.576+0000 7fbc65727200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:10.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:10.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:10.751+0000 7f97c4233200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:11.011 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:11.267 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:11.009+0000 7f97c4233200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:52:11.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:11.241+0000 7f97c4233200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:52:11.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:11.561+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 14) ==== 91699+0+0 (secure 0 0 0) 0x7f20c4023490 con 0x7f20d4101200 2024-03-18T15:52:11.808 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: Standby manager daemon z started 2024-03-18T15:52:11.808 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:52:11.808 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]: dispatch 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]': finished 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:52:11.809 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:11 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: Standby manager daemon z started 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]: dispatch 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]': finished 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:52:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:11 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:12.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: Standby manager daemon z started 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/3410408653' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]: dispatch 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3813553822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d33d5d-faae-42ac-b7d3-70679106ee85"}]': finished 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T15:52:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:11 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:12.517 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:12.029+0000 7f97c4233200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:52:12.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:12 smithi150 ceph-mon[31946]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:12 smithi150 ceph-mon[31946]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-18T15:52:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:52:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:12 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3072061315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:12.902 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-mon[35919]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:12.902 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-mon[35919]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-18T15:52:12.902 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:52:12.903 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3072061315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:12.903 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:12.594+0000 7f97c4233200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:52:12.903 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:12.690+0000 7f97c4233200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:52:12.903 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:52:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:52:12.901+0000 7f97c4233200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:52:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:12 smithi191 ceph-mon[36784]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:12 smithi191 ceph-mon[36784]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-18T15:52:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T15:52:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:12 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3072061315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:13.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:13.568+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mon.0 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f20c40092c0 con 0x7f20d4101200 2024-03-18T15:52:13.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:13 smithi150 ceph-mon[31946]: Standby manager daemon y started 2024-03-18T15:52:13.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:13 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:52:13.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:13 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:13 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:52:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:13 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:13 smithi191 ceph-mon[36784]: Standby manager daemon y started 2024-03-18T15:52:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:13 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:52:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:13 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:13 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:52:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:13 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:13 smithi156 ceph-mon[35919]: Standby manager daemon y started 2024-03-18T15:52:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:13 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T15:52:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:13 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T15:52:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:13 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T15:52:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:13 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/2401294198' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T15:52:14.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:14 smithi150 ceph-mon[31946]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:14.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:14 smithi150 ceph-mon[31946]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-18T15:52:14.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:52:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:14 smithi191 ceph-mon[36784]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:14 smithi191 ceph-mon[36784]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-18T15:52:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:52:15.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:14 smithi156 ceph-mon[35919]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:14 smithi156 ceph-mon[35919]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-03-18T15:52:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T15:52:15.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:15 smithi150 ceph-mon[31946]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:15.961 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:15 smithi191 ceph-mon[36784]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:16.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:15 smithi156 ceph-mon[35919]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:18.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:18 smithi150 ceph-mon[31946]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:18.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:18 smithi191 ceph-mon[36784]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:18.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:18 smithi156 ceph-mon[35919]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:19.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:52:19.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:19.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:19 smithi191 ceph-mon[36784]: Deploying daemon osd.0 on smithi150 2024-03-18T15:52:19.732 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:52:19.732 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:19.732 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:19 smithi150 ceph-mon[31946]: Deploying daemon osd.0 on smithi150 2024-03-18T15:52:19.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T15:52:19.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:19.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:19 smithi156 ceph-mon[35919]: Deploying daemon osd.0 on smithi150 2024-03-18T15:52:20.672 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:20 smithi150 ceph-mon[31946]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:20.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:20 smithi191 ceph-mon[36784]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:20 smithi156 ceph-mon[35919]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:22.577 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:22 smithi150 ceph-mon[31946]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:22.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:22 smithi191 ceph-mon[36784]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:22 smithi156 ceph-mon[35919]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:24.708 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:24 smithi156 ceph-mon[35919]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:24.708 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.708 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.708 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:24.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:24 smithi191 ceph-mon[36784]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:24.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:24.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:24 smithi150 ceph-mon[31946]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:26.656 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:26 smithi150 ceph-mon[31946]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:26 smithi191 ceph-mon[36784]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:26.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:26 smithi156 ceph-mon[35919]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:27.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.024+0000 7f20d17fa700 1 -- 172.21.15.150:0/3371164419 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f20d41a2f00 con 0x7f20c0039200 2024-03-18T15:52:27.026 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 0 on host 'smithi150' 2024-03-18T15:52:27.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 msgr2=0x7f20c003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 0x7f20c003b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20bc007be0 tx=0x7f20bc0073d0 comp rx=0 tx=0).stop 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 msgr2=0x7f20d4105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f20c400d980 tx=0x7f20c400dc90 comp rx=0 tx=0).stop 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 shutdown_connections 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f20c0039200 0x7f20c003b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20d4102da0 0x7f20d4103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d4101200 0x7f20d4105190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 --2- 172.21.15.150:0/3371164419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d4102400 0x7f20d41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 >> 172.21.15.150:0/3371164419 conn(0x7f20d40fc7a0 msgr2=0x7f20d40fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:52:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 shutdown_connections 2024-03-18T15:52:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:52:27.026+0000 7f20d9de7700 1 -- 172.21.15.150:0/3371164419 wait complete. 2024-03-18T15:52:27.650 DEBUG:teuthology.orchestra.run.smithi150:osd.0> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T15:52:27.693 INFO:tasks.cephadm:Deploying osd.1 on smithi156 with /dev/vg_nvme/lv_4... 2024-03-18T15:52:27.693 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:52:27.942 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:52:28.265 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.266 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.266 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:28.266 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:52:28.266 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.266 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:27 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.316 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.316 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.317 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:28.317 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:52:28.317 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.317 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:27 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:27 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:29.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:29 smithi156 ceph-mon[35919]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:52:29.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:29 smithi156 ceph-mon[35919]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:52:29.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:29 smithi156 ceph-mon[35919]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:52:29.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:29.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-mon[31946]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:52:29.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-mon[31946]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:52:29.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-mon[31946]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:52:29.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:29 smithi191 ceph-mon[36784]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T15:52:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:29 smithi191 ceph-mon[36784]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T15:52:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:29 smithi191 ceph-mon[36784]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:52:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:29.604 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0[44221]: 2024-03-18T15:52:29.410+0000 7fcb27066700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T15:52:29.604 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:52:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0[44221]: 2024-03-18T15:52:29.415+0000 7fcb1ee51700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0083523 s, 1.3 GB/s 2024-03-18T15:52:29.769 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-18T15:52:30.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:52:30.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:52:30.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:30 smithi150 ceph-mon[31946]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:30.342 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:30 smithi156 ceph-mon[35919]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' 2024-03-18T15:52:30.367 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:52:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:30 smithi191 ceph-mon[36784]: from='osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622]' entity='osd.0' 2024-03-18T15:52:30.605 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] boot 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:52:31.254 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:31.267 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:52:31.267 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:52:31.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:31.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] boot 2024-03-18T15:52:31.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:52:31.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: osd.0 [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] boot 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T15:52:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T15:52:32.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 -- 172.21.15.156:0/2864948552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 msgr2=0x7f4750100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 --2- 172.21.15.156:0/2864948552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f4750100a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f474c009990 tx=0x7f474c009ca0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 -- 172.21.15.156:0/2864948552 shutdown_connections 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 --2- 172.21.15.156:0/2864948552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101d40 0x7f47501065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 --2- 172.21.15.156:0/2864948552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f4750100a30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 --2- 172.21.15.156:0/2864948552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4750101380 0x7f4750101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 -- 172.21.15.156:0/2864948552 >> 172.21.15.156:0/2864948552 conn(0x7f47500fbf70 msgr2=0x7f47500fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 -- 172.21.15.156:0/2864948552 shutdown_connections 2024-03-18T15:52:32.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 -- 172.21.15.156:0/2864948552 wait complete. 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.061+0000 7f4757d68700 1 Processor -- start 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- start start 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 0x7f475019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4750101d40 0x7f47501a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:32.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4750114570 con 0x7f4750101380 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47501143f0 con 0x7f4750100610 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f47501146f0 con 0x7f4750101d40 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59916/0 (socket says 172.21.15.156:59916) 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 -- 172.21.15.156:0/4119503095 learned_addr learned my addr 172.21.15.156:0/4119503095 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755303700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 0x7f475019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 -- 172.21.15.156:0/4119503095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4750101d40 msgr2=0x7f47501a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4750101d40 0x7f47501a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 -- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 msgr2=0x7f475019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 0x7f475019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 -- 172.21.15.156:0/4119503095 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47501a6720 con 0x7f4750100610 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755303700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 0x7f475019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4755b04700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f474c005910 tx=0x7f474c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:32.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c027070 con 0x7f4750100610 2024-03-18T15:52:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47501a69b0 con 0x7f4750100610 2024-03-18T15:52:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474c004210 con 0x7f4750100610 2024-03-18T15:52:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c00f460 con 0x7f4750100610 2024-03-18T15:52:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.062+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47501a6e00 con 0x7f4750100610 2024-03-18T15:52:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.063+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4750068f90 con 0x7f4750100610 2024-03-18T15:52:32.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.064+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f474c02a070 con 0x7f4750100610 2024-03-18T15:52:32.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.065+0000 7f4746ffd700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 0x7f473c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:32.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.065+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f474c0ce670 con 0x7f4750100610 2024-03-18T15:52:32.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.065+0000 7f4755303700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 0x7f473c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:32.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.066+0000 7f4755303700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 0x7f473c0a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f474000bcf0 tx=0x7f474000b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:32.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.069+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f474c031070 con 0x7f4750100610 2024-03-18T15:52:32.278 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:32.278+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4750062710 con 0x7f473c0a2800 2024-03-18T15:52:32.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi150 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 17088M 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:32.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:32.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:32.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:32 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi150 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 17088M 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:32.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:32 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi150 2024-03-18T15:52:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 17088M 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:32.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:32 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:33 smithi156 ceph-mon[35919]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:33.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:33 smithi150 ceph-mon[31946]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:33 smithi191 ceph-mon[36784]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:34.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:34 smithi150 ceph-mon[31946]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:34 smithi191 ceph-mon[36784]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:34.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:34 smithi156 ceph-mon[35919]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:35.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:35 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/1298526422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:35 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:35 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]': finished 2024-03-18T15:52:35.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:35 smithi150 ceph-mon[31946]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:52:35.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:35 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:35.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:35 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/1298526422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:35 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:35 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]': finished 2024-03-18T15:52:35.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:35 smithi191 ceph-mon[36784]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:52:35.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:35 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:35 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/1298526422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:35 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]: dispatch 2024-03-18T15:52:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:35 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14890f1b-f4e5-4d2d-a1f0-71c84f464fd0"}]': finished 2024-03-18T15:52:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:35 smithi156 ceph-mon[35919]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T15:52:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:35 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:36.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:36 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/3665558150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:36.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:36 smithi150 ceph-mon[31946]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:36.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:36 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/3665558150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:36.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:36 smithi191 ceph-mon[36784]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:36.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:36 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/3665558150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:36.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:36 smithi156 ceph-mon[35919]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:38.671 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:38 smithi156 ceph-mon[35919]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:38.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:38 smithi191 ceph-mon[36784]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:38.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:38 smithi150 ceph-mon[31946]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:40.685 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:40 smithi150 ceph-mon[31946]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:40.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:40 smithi191 ceph-mon[36784]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:40 smithi156 ceph-mon[35919]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:42.671 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:42 smithi156 ceph-mon[35919]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:42.671 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:52:42.671 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:42.671 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:42 smithi156 ceph-mon[35919]: Deploying daemon osd.1 on smithi156 2024-03-18T15:52:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:42 smithi191 ceph-mon[36784]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:52:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:42 smithi191 ceph-mon[36784]: Deploying daemon osd.1 on smithi156 2024-03-18T15:52:42.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:42 smithi150 ceph-mon[31946]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:42.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T15:52:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:42 smithi150 ceph-mon[31946]: Deploying daemon osd.1 on smithi156 2024-03-18T15:52:44.666 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:44 smithi156 ceph-mon[35919]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:44.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:44 smithi191 ceph-mon[36784]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:44.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:44 smithi150 ceph-mon[31946]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:46.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:46 smithi191 ceph-mon[36784]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:46.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:46 smithi156 ceph-mon[35919]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:46.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:46 smithi150 ceph-mon[31946]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:47 smithi156 ceph-mon[35919]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:48.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:47 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:47 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:47 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:48.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:47 smithi150 ceph-mon[31946]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:48.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:47 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:47 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:48.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:47 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:48.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:47 smithi191 ceph-mon[36784]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:49.990 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 1 on host 'smithi156' 2024-03-18T15:52:49.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.989+0000 7f4746ffd700 1 -- 172.21.15.156:0/4119503095 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4750062710 con 0x7f473c0a2800 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 msgr2=0x7f473c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 0x7f473c0a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f474000bcf0 tx=0x7f474000b2c0 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 msgr2=0x7f475019eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f474c005910 tx=0x7f474c009580 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 shutdown_connections 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4750101d40 0x7f47501a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f473c0a2800 0x7f473c0a4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4750100610 0x7f475019eac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 --2- 172.21.15.156:0/4119503095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4750101380 0x7f475019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 >> 172.21.15.156:0/4119503095 conn(0x7f47500fbf70 msgr2=0x7f47501025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 shutdown_connections 2024-03-18T15:52:49.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:52:49.992+0000 7f4757d68700 1 -- 172.21.15.156:0/4119503095 wait complete. 2024-03-18T15:52:50.565 DEBUG:teuthology.orchestra.run.smithi156:osd.1> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T15:52:50.567 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/vg_nvme/lv_4... 2024-03-18T15:52:50.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T15:52:50.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-mon[35919]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:50.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.652 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.652 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.652 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:50 smithi191 ceph-mon[36784]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.814 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:52:50.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:50 smithi150 ceph-mon[31946]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:50.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:50.926 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:52:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T15:52:50.650+0000 7fad31a327c0 -1 osd.1 0 log_to_monitors true 2024-03-18T15:52:51.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:51 smithi156 ceph-mon[35919]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:51.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:51 smithi156 ceph-mon[35919]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:51.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:51 smithi191 ceph-mon[36784]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:51.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:51 smithi191 ceph-mon[36784]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:51.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:51 smithi150 ceph-mon[31946]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:51.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:51 smithi150 ceph-mon[31946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111259 s, 942 MB/s 2024-03-18T15:52:52.625 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:52 smithi191 ceph-mon[36784]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:52.712 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:52:52.713 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:52.713 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:52:52.713 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:52.713 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-mon[35919]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:52.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T15:52:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: from='osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T15:52:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:52 smithi150 ceph-mon[31946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:52:53.011 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T15:52:52.932+0000 7fad27097700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T15:52:53.012 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:52:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T15:52:52.936+0000 7fad1ee82700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:52:53.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-03-18T15:52:53.465 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:52:53.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:52:53.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:52:53.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.586 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='osd.1 ' entity='osd.1' 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:53.587 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:53 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='osd.1 ' entity='osd.1' 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:53.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:53 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='osd.1 ' entity='osd.1' 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:53 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:54.691 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi156 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:54.692 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] boot 2024-03-18T15:52:54.693 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:52:54.693 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi156 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] boot 2024-03-18T15:52:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:52:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi156 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T15:52:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: osd.1 [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] boot 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T15:52:54.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2769996568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc1056c0 msgr2=0x7fe0cc107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2769996568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc107ab0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009990 tx=0x7fe0c0009ca0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2769996568 shutdown_connections 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2769996568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 0x7fe0cc10a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2769996568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc102d90 0x7fe0cc105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2769996568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc107ab0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2769996568 >> 172.21.15.191:0/2769996568 conn(0x7fe0cc0fc7c0 msgr2=0x7fe0cc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:52:55.037 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2769996568 shutdown_connections 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2769996568 wait complete. 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 Processor -- start 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.030+0000 7fe0d1c71700 1 -- start start 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 0x7fe0cc19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc10df50 con 0x7fe0cc107ff0 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc10ddd0 con 0x7fe0cc1056c0 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc10e0d0 con 0x7fe0cc102d90 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:55222/0 (socket says 172.21.15.191:55222) 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 -- 172.21.15.191:0/2095858798 learned_addr learned my addr 172.21.15.191:0/2095858798 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:52:55.038 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0caffd700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cbfff700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 0x7fe0cc19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 -- 172.21.15.191:0/2095858798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc1056c0 msgr2=0x7fe0cc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 -- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 msgr2=0x7fe0cc19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 0x7fe0cc19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 -- 172.21.15.191:0/2095858798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc1a0220 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0cb7fe700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00b560 tx=0x7fe0bc00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc015020 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0bc011d10 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc1a0510 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc010e50 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.031+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc1a0ad0 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.032+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc068f90 con 0x7fe0cc102d90 2024-03-18T15:52:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.033+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7fe0bc018020 con 0x7fe0cc102d90 2024-03-18T15:52:55.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.034+0000 7fe0c8ff9700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 0x7fe0b40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:52:55.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.034+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fe0bc0cc980 con 0x7fe0cc102d90 2024-03-18T15:52:55.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.034+0000 7fe0caffd700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 0x7fe0b40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:52:55.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.035+0000 7fe0caffd700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 0x7fe0b40a4e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000b5c0 tx=0x7fe0c0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:52:55.040 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.039+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0bc01f020 con 0x7fe0cc102d90 2024-03-18T15:52:55.244 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:52:55.243+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe0cc062710 con 0x7fe0b40a29a0 2024-03-18T15:52:55.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:55.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:55.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:55.714 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:55.714 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:55.714 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:55.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:52:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:52:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:52:56.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:56 smithi191 ceph-mon[36784]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:56.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:56 smithi191 ceph-mon[36784]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:56.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:56 smithi191 ceph-mon[36784]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:52:56.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:56 smithi156 ceph-mon[35919]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:56.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:56 smithi156 ceph-mon[35919]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:56.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:56 smithi156 ceph-mon[35919]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:52:56.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:56 smithi150 ceph-mon[31946]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:52:56.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:56 smithi150 ceph-mon[31946]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:56.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:56 smithi150 ceph-mon[31946]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T15:52:58.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:58.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/3776588365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]': finished 2024-03-18T15:52:58.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:52:58.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:58 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/3776588365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]': finished 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:52:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:58 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:52:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:52:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/3776588365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]: dispatch 2024-03-18T15:52:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f5a563a-5180-40bd-a2a9-625033da15e4"}]': finished 2024-03-18T15:52:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T15:52:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:58 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:52:59.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:52:59 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/205027642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:59.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:52:59 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/205027642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:52:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:52:59 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/205027642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:00.758 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:00 smithi150 ceph-mon[31946]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:00.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:00 smithi156 ceph-mon[35919]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:00.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:00 smithi191 ceph-mon[36784]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:02.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:02 smithi191 ceph-mon[36784]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:02.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:02 smithi156 ceph-mon[35919]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:02.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:02 smithi150 ceph-mon[31946]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:04.731 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:04 smithi191 ceph-mon[36784]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:04.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:04 smithi156 ceph-mon[35919]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:04.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:04 smithi150 ceph-mon[31946]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:05.760 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:05 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:53:05.760 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:05 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:05 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:53:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:05 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:05.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:05 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T15:53:05.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:05 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:06.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:06 smithi156 ceph-mon[35919]: Deploying daemon osd.2 on smithi191 2024-03-18T15:53:06.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:06 smithi156 ceph-mon[35919]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:06 smithi191 ceph-mon[36784]: Deploying daemon osd.2 on smithi191 2024-03-18T15:53:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:06 smithi191 ceph-mon[36784]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:06.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:06 smithi150 ceph-mon[31946]: Deploying daemon osd.2 on smithi191 2024-03-18T15:53:06.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:06 smithi150 ceph-mon[31946]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:08.759 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:08 smithi191 ceph-mon[36784]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:08 smithi156 ceph-mon[35919]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:08.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:08 smithi150 ceph-mon[31946]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:10 smithi156 ceph-mon[35919]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:10.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:10 smithi150 ceph-mon[31946]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:10.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:10.866 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:10 smithi191 ceph-mon[36784]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:10.866 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.866 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:10.866 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:12.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:12 smithi191 ceph-mon[36784]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:12.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:12 smithi156 ceph-mon[35919]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:12.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:12 smithi150 ceph-mon[31946]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:13.507 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2024-03-18T15:53:13.508 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.506+0000 7fe0c8ff9700 1 -- 172.21.15.191:0/2095858798 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe0cc062710 con 0x7fe0b40a29a0 2024-03-18T15:53:13.510 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.508+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 msgr2=0x7fe0b40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.508+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 0x7fe0b40a4e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000b5c0 tx=0x7fe0c0005fd0 comp rx=0 tx=0).stop 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 msgr2=0x7fe0cc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00b560 tx=0x7fe0bc00b870 comp rx=0 tx=0).stop 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 shutdown_connections 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0cc107ff0 0x7fe0cc19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe0cc102d90 0x7fe0cc1985c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe0b40a29a0 0x7fe0b40a4e60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:13.511 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 --2- 172.21.15.191:0/2095858798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe0cc1056c0 0x7fe0cc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:13.512 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 >> 172.21.15.191:0/2095858798 conn(0x7fe0cc0fc7c0 msgr2=0x7fe0cc103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:53:13.512 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 shutdown_connections 2024-03-18T15:53:13.512 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:53:13.509+0000 7fe0d1c71700 1 -- 172.21.15.191:0/2095858798 wait complete. 2024-03-18T15:53:14.137 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T15:53:14.140 INFO:tasks.cephadm:Deploying osd.3 on smithi150 with /dev/vg_nvme/lv_3... 2024-03-18T15:53:14.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:53:14.392 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.821 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.828 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:14.828 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T15:53:14.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:14.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:15.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:15 smithi156 ceph-mon[35919]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:53:15.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:15 smithi156 ceph-mon[35919]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:53:15.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:15 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:15.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:15 smithi156 ceph-mon[35919]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:15.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:15 smithi156 ceph-mon[35919]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:15.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:15 smithi150 ceph-mon[31946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:53:15.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:15 smithi150 ceph-mon[31946]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:53:15.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:15 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:15.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:15 smithi150 ceph-mon[31946]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:15.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:15 smithi150 ceph-mon[31946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:15.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-mon[36784]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T15:53:15.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-mon[36784]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T15:53:15.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:15.821 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-mon[36784]: from='osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:15.822 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-mon[36784]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:53:16.122 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2[41949]: 2024-03-18T15:53:15.909+0000 7f7516348700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T15:53:16.122 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2[41949]: 2024-03-18T15:53:15.912+0000 7f750e133700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144508 s, 726 MB/s 2024-03-18T15:53:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.687 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:16 smithi150 ceph-mon[31946]: from='osd.2 ' entity='osd.2' 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:16 smithi156 ceph-mon[35919]: from='osd.2 ' entity='osd.2' 2024-03-18T15:53:16.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:16 smithi191 ceph-mon[36784]: from='osd.2 ' entity='osd.2' 2024-03-18T15:53:17.197 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] boot 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:53:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:17.981 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:53:17.982 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:53:17.982 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:17.982 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] boot 2024-03-18T15:53:17.982 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:53:17.982 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: osd.2 [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] boot 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T15:53:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 -- 172.21.15.150:0/640397827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674100380 msgr2=0x7fa674100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 --2- 172.21.15.150:0/640397827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674100380 0x7fa674100800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa6700099c0 tx=0x7fa670009cd0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 -- 172.21.15.150:0/640397827 shutdown_connections 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 --2- 172.21.15.150:0/640397827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 0x7fa67410a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 --2- 172.21.15.150:0/640397827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674103900 0x7fa674103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 --2- 172.21.15.150:0/640397827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674100380 0x7fa674100800 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 -- 172.21.15.150:0/640397827 >> 172.21.15.150:0/640397827 conn(0x7fa6740fbf40 msgr2=0x7fa6740fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:53:18.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 -- 172.21.15.150:0/640397827 shutdown_connections 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.806+0000 7fa67b769700 1 -- 172.21.15.150:0/640397827 wait complete. 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 Processor -- start 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 -- start start 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674100380 0x7fa674198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 0x7fa674198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410df10 con 0x7fa674103900 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410dd90 con 0x7fa674100d40 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa67b769700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410e090 con 0x7fa674100380 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43702/0 (socket says 172.21.15.150:43702) 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 -- 172.21.15.150:0/245021218 learned_addr learned my addr 172.21.15.150:0/245021218 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:53:18.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679505700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674100380 0x7fa674198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa678d04700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 0x7fa674198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 -- 172.21.15.150:0/245021218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674100380 msgr2=0x7fa674198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674100380 0x7fa674198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 -- 172.21.15.150:0/245021218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 msgr2=0x7fa674198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 0x7fa674198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.807+0000 7fa679d06700 1 -- 172.21.15.150:0/245021218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6741a01c0 con 0x7fa674103900 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa679d06700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00add0 tx=0x7fa66c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c017020 con 0x7fa674103900 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66c00ae00 con 0x7fa674103900 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6741a04b0 con 0x7fa674103900 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0053e0 con 0x7fa674103900 2024-03-18T15:53:18.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.808+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6741a09f0 con 0x7fa674103900 2024-03-18T15:53:18.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.809+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa674068f90 con 0x7fa674103900 2024-03-18T15:53:18.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.810+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7fa66c00c890 con 0x7fa674103900 2024-03-18T15:53:18.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.810+0000 7fa66a7fc700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 0x7fa6600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:18.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.811+0000 7fa679505700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 0x7fa6600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:18.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.811+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fa66c0cd2a0 con 0x7fa674103900 2024-03-18T15:53:18.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.811+0000 7fa679505700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 0x7fa6600a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa664005950 tx=0x7fa66400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:53:18.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:18.814+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa66c021020 con 0x7fa674103900 2024-03-18T15:53:18.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:18.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi191 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:18.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi191 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi191 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 17080M 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:19.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:19.040+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa6740627d0 con 0x7fa6600a2800 2024-03-18T15:53:19.342 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[48978]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:19.342 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[48978]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:19.342 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[48978]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:19.342 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[48978]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:19.784 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[49021]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:19.784 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[49021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:19.784 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[49021]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:19.784 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 15:53:19 smithi150 sudo[49021]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:20.056 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:19 smithi156 sudo[44540]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:20.056 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:19 smithi156 sudo[44540]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:20.056 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:19 smithi156 sudo[44540]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:20.056 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:19 smithi156 sudo[44540]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:20.056 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:20.057 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:20.057 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:20.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:20.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:20.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:20.518 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:20 smithi156 sudo[44543]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:20.518 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:20 smithi156 sudo[44543]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:20.518 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:20 smithi156 sudo[44543]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:20.518 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 15:53:20 smithi156 sudo[44543]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:20.575 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46031]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:20.575 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46031]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:20.575 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46031]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:20.575 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46031]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:20.962 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46034]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:20.962 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46034]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:20.962 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46034]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:20.962 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 15:53:20 smithi191 sudo[46034]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:21.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:20 smithi150 sudo[49079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:21.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:20 smithi150 sudo[49079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:21.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:20 smithi150 sudo[49079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:21.066 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:20 smithi150 sudo[49079]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:21.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:21.128+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa66c097800 con 0x7fa674103900 2024-03-18T15:53:21.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 sudo[49086]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 sudo[49086]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 sudo[49086]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 sudo[49086]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 ceph-mon[31946]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:21 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 ceph-mon[36784]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46037]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46037]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46037]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:21.378 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46037]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:21.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 ceph-mon[35919]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T15:53:21.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:21.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:21.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46040]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:21.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46040]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:21.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46040]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:21.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:21 smithi191 sudo[46040]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 sudo[44546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T15:53:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 sudo[44546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 sudo[44546]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:22.108 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:21 smithi156 sudo[44546]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.456 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]: dispatch 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]': finished 2024-03-18T15:53:22.457 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:53:22.458 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]: dispatch 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]': finished 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:53:22.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 sudo[44549]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 sudo[44549]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 sudo[44549]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 sudo[44549]: pam_unix(sudo:session): session closed for user root 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]: dispatch 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T15:53:22.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/983500440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "869ada62-adef-48b5-8d36-f57fb0220c2a"}]': finished 2024-03-18T15:53:22.520 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T15:53:22.520 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:23.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:23 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3145565415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:23.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:23 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3145565415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:23.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:23 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3145565415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:24.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:24 smithi191 ceph-mon[36784]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:24.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:24 smithi156 ceph-mon[35919]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:24.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:24 smithi150 ceph-mon[31946]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:26 smithi191 ceph-mon[36784]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 335 KiB/s wr, 23 op/s 2024-03-18T15:53:26.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:26 smithi156 ceph-mon[35919]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 335 KiB/s wr, 23 op/s 2024-03-18T15:53:26.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:26 smithi150 ceph-mon[31946]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 335 KiB/s wr, 23 op/s 2024-03-18T15:53:28.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:28 smithi150 ceph-mon[31946]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 245 KiB/s wr, 16 op/s 2024-03-18T15:53:28.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:28 smithi191 ceph-mon[36784]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 245 KiB/s wr, 16 op/s 2024-03-18T15:53:28.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:28 smithi156 ceph-mon[35919]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 245 KiB/s wr, 16 op/s 2024-03-18T15:53:29.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:53:29.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:29.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:53:29.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:29.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T15:53:29.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:30.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:30 smithi150 ceph-mon[31946]: Deploying daemon osd.3 on smithi150 2024-03-18T15:53:30.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:30 smithi150 ceph-mon[31946]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 193 KiB/s wr, 13 op/s 2024-03-18T15:53:30.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:30 smithi191 ceph-mon[36784]: Deploying daemon osd.3 on smithi150 2024-03-18T15:53:30.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:30 smithi191 ceph-mon[36784]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 193 KiB/s wr, 13 op/s 2024-03-18T15:53:30.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:30 smithi156 ceph-mon[35919]: Deploying daemon osd.3 on smithi150 2024-03-18T15:53:30.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:30 smithi156 ceph-mon[35919]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 193 KiB/s wr, 13 op/s 2024-03-18T15:53:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:32 smithi191 ceph-mon[36784]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T15:53:32.719 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:32 smithi150 ceph-mon[31946]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T15:53:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:32 smithi156 ceph-mon[35919]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T15:53:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:34 smithi191 ceph-mon[36784]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-03-18T15:53:34.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:34 smithi156 ceph-mon[35919]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-03-18T15:53:34.773 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:34 smithi150 ceph-mon[31946]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-03-18T15:53:36.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:36.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:36 smithi156 ceph-mon[35919]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T15:53:36.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:36.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:36 smithi150 ceph-mon[31946]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T15:53:36.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:36.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:36.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:36 smithi191 ceph-mon[36784]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T15:53:38.470 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:38 smithi150 ceph-mon[31946]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:38.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.549+0000 7fa66a7fc700 1 -- 172.21.15.150:0/245021218 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa6740627d0 con 0x7fa6600a2800 2024-03-18T15:53:38.550 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 3 on host 'smithi150' 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 msgr2=0x7fa6600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 0x7fa6600a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa664005950 tx=0x7fa66400a300 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 msgr2=0x7fa67419fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00add0 tx=0x7fa66c00c330 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 shutdown_connections 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa674103900 0x7fa67419fb20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa674100380 0x7fa674198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa6600a2800 0x7fa6600a4cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 --2- 172.21.15.150:0/245021218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa674100d40 0x7fa674198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 >> 172.21.15.150:0/245021218 conn(0x7fa6740fbf40 msgr2=0x7fa6741069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:53:38.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 shutdown_connections 2024-03-18T15:53:38.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:53:38.552+0000 7fa67b769700 1 -- 172.21.15.150:0/245021218 wait complete. 2024-03-18T15:53:38.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:38 smithi191 ceph-mon[36784]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:38.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:38 smithi156 ceph-mon[35919]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:39.253 DEBUG:teuthology.orchestra.run.smithi150:osd.3> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T15:53:39.255 INFO:tasks.cephadm:Deploying osd.4 on smithi156 with /dev/vg_nvme/lv_3... 2024-03-18T15:53:39.255 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:53:39.502 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:53:39.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:39 smithi156 ceph-mon[35919]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:53:39.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:39 smithi150 ceph-mon[31946]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:53:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:39 smithi191 ceph-mon[36784]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T15:53:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:40 smithi156 ceph-mon[35919]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:40 smithi156 ceph-mon[35919]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:53:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:40 smithi156 ceph-mon[35919]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:53:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:40 smithi156 ceph-mon[35919]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:53:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:40 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:40.776 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:40 smithi150 ceph-mon[31946]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:40.776 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:40 smithi150 ceph-mon[31946]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:53:40.776 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:40 smithi150 ceph-mon[31946]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:53:40.776 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:40 smithi150 ceph-mon[31946]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:53:40.776 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:40 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:40 smithi191 ceph-mon[36784]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:40 smithi191 ceph-mon[36784]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T15:53:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:40 smithi191 ceph-mon[36784]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T15:53:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:40 smithi191 ceph-mon[36784]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:53:40.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:40 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00867156 s, 1.2 GB/s 2024-03-18T15:53:41.398 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-18T15:53:41.592 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3[51166]: 2024-03-18T15:53:41.334+0000 7f8c15577700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T15:53:41.593 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3[51166]: 2024-03-18T15:53:41.340+0000 7f8c0d362700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:53:42.005 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:53:42.006 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:53:42.006 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.006 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.006 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' 2024-03-18T15:53:42.006 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:41 smithi156 ceph-mon[35919]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:42.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' 2024-03-18T15:53:42.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:41 smithi191 ceph-mon[36784]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:42.272 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:53:42.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:53:42.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T15:53:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: from='osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380]' entity='osd.3' 2024-03-18T15:53:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:41 smithi150 ceph-mon[31946]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] boot 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:42.977 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:42.978 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:42.978 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:42.978 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] boot 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:43.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:43.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: osd.3 [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] boot 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 -- 172.21.15.156:0/3749387601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f0102d80 msgr2=0x7fb8f0105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 --2- 172.21.15.156:0/3749387601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f0102d80 0x7fb8f0105170 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00b0a0 tx=0x7fb8ec00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 -- 172.21.15.156:0/3749387601 shutdown_connections 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 --2- 172.21.15.156:0/3749387601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0107fe0 0x7fb8f010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 --2- 172.21.15.156:0/3749387601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f0102d80 0x7fb8f0105170 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 --2- 172.21.15.156:0/3749387601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f01056b0 0x7fb8f0107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 -- 172.21.15.156:0/3749387601 >> 172.21.15.156:0/3749387601 conn(0x7fb8f00fc790 msgr2=0x7fb8f00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 -- 172.21.15.156:0/3749387601 shutdown_connections 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.749+0000 7fb8f7514700 1 -- 172.21.15.156:0/3749387601 wait complete. 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 Processor -- start 2024-03-18T15:53:43.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 -- start start 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0102d80 0x7fb8f01026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 0x7fb8f0101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010dfc0 con 0x7fb8f0107fe0 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010de40 con 0x7fb8f01056b0 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f7514700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010e140 con 0x7fb8f0102d80 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52666/0 (socket says 172.21.15.156:52666) 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 -- 172.21.15.156:0/2572647908 learned_addr learned my addr 172.21.15.156:0/2572647908 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f5ab1700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 0x7fb8f0101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f52b0700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0102d80 0x7fb8f01026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 -- 172.21.15.156:0/2572647908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0102d80 msgr2=0x7fb8f01026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0102d80 0x7fb8f01026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 -- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 msgr2=0x7fb8f0101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 0x7fb8f0101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 -- 172.21.15.156:0/2572647908 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f0101b20 con 0x7fb8f01056b0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f5ab1700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 0x7fb8f0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.750+0000 7fb8f4aaf700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000b530 tx=0x7fb8e000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.751+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0015020 con 0x7fb8f01056b0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.751+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f01a2740 con 0x7fb8f01056b0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.751+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e0011d10 con 0x7fb8f01056b0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.751+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0010e50 con 0x7fb8f01056b0 2024-03-18T15:53:43.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.751+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f01a2d00 con 0x7fb8f01056b0 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.752+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8f0068f90 con 0x7fb8f01056b0 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.753+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb8e0018020 con 0x7fb8f01056b0 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.754+0000 7fb8e67fc700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 0x7fb8dc0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.754+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3698+0+0 (secure 0 0 0) 0x7fb8e00cd6c0 con 0x7fb8f01056b0 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.754+0000 7fb8f52b0700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 0x7fb8dc0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:53:43.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.755+0000 7fb8f52b0700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 0x7fb8dc0a4ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec007f00 tx=0x7fb8ec006230 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:53:43.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.758+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb8e001f020 con 0x7fb8f01056b0 2024-03-18T15:53:43.965 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:53:43.964+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb8f0062710 con 0x7fb8dc0a2a30 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi150 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 8544M 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:44.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:43 smithi191 ceph-mon[36784]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi150 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 8544M 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:44.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:43 smithi156 ceph-mon[35919]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:53:44.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi150 2024-03-18T15:53:44.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 8544M 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:44.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:43 smithi150 ceph-mon[31946]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T15:53:45.211 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:44 smithi156 ceph-mon[35919]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:45.211 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:45.211 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:45.211 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:45.211 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:44 smithi156 ceph-mon[35919]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:53:45.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:44 smithi191 ceph-mon[36784]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:45.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:45.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:45.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:45.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:44 smithi191 ceph-mon[36784]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:53:45.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:44 smithi150 ceph-mon[31946]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:53:45.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:53:45.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:53:45.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:45.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:44 smithi150 ceph-mon[31946]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T15:53:46.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:46 smithi156 ceph-mon[35919]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:53:46.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:46 smithi156 ceph-mon[35919]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-18T15:53:46.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:46 smithi150 ceph-mon[31946]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:53:46.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:46 smithi150 ceph-mon[31946]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-18T15:53:46.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:46 smithi191 ceph-mon[36784]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T15:53:46.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:46 smithi191 ceph-mon[36784]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 126 MiB used, 357 GiB / 358 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-18T15:53:47.517 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/4286722389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]': finished 2024-03-18T15:53:47.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:53:47.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:53:47.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/3033902605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:47.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/4286722389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]': finished 2024-03-18T15:53:47.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:53:47.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:53:47.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/3033902605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/4286722389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]: dispatch 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7324a8d6-300d-4163-8282-c368870e2bfd"}]': finished 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:53:47.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/3033902605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:53:48.517 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:48 smithi156 ceph-mon[35919]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:53:48.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:48 smithi150 ceph-mon[31946]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:53:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:48 smithi191 ceph-mon[36784]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:53:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:50 smithi191 ceph-mon[36784]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T15:53:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:50 smithi156 ceph-mon[35919]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T15:53:50.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:50 smithi150 ceph-mon[31946]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T15:53:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:52 smithi191 ceph-mon[36784]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:53:52.738 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:52 smithi156 ceph-mon[35919]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:53:52.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:52 smithi150 ceph-mon[31946]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:53:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:54 smithi156 ceph-mon[35919]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:53:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:54 smithi191 ceph-mon[36784]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:53:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:54.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:54 smithi150 ceph-mon[31946]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T15:53:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:53:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:55 smithi156 ceph-mon[35919]: Deploying daemon osd.4 on smithi156 2024-03-18T15:53:55.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:55 smithi191 ceph-mon[36784]: Deploying daemon osd.4 on smithi156 2024-03-18T15:53:55.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:55 smithi150 ceph-mon[31946]: Deploying daemon osd.4 on smithi156 2024-03-18T15:53:56.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:56 smithi191 ceph-mon[36784]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:56.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:56 smithi156 ceph-mon[35919]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:56.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:56 smithi150 ceph-mon[31946]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:58.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:58 smithi156 ceph-mon[35919]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:58.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:58 smithi191 ceph-mon[36784]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:53:58.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:58 smithi150 ceph-mon[31946]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:53:59 smithi156 ceph-mon[35919]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:00.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:00.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:53:59 smithi191 ceph-mon[36784]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:00.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:00.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:00.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:53:59 smithi150 ceph-mon[31946]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:02.577 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:02 smithi156 ceph-mon[35919]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:02.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.576+0000 7fb8e67fc700 1 -- 172.21.15.156:0/2572647908 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb8f0062710 con 0x7fb8dc0a2a30 2024-03-18T15:54:02.577 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 4 on host 'smithi156' 2024-03-18T15:54:02.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.578+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 msgr2=0x7fb8dc0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:02.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.578+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 0x7fb8dc0a4ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec007f00 tx=0x7fb8ec006230 comp rx=0 tx=0).stop 2024-03-18T15:54:02.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.578+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 msgr2=0x7fb8f0100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.578+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000b530 tx=0x7fb8e000b8f0 comp rx=0 tx=0).stop 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 shutdown_connections 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8f0107fe0 0x7fb8f0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8f0102d80 0x7fb8f01026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8dc0a2a30 0x7fb8dc0a4ef0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 --2- 172.21.15.156:0/2572647908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f01056b0 0x7fb8f0100d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 >> 172.21.15.156:0/2572647908 conn(0x7fb8f00fc790 msgr2=0x7fb8f0103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 shutdown_connections 2024-03-18T15:54:02.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:02.579+0000 7fb8f7514700 1 -- 172.21.15.156:0/2572647908 wait complete. 2024-03-18T15:54:02.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:02 smithi191 ceph-mon[36784]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:02.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:02 smithi150 ceph-mon[31946]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:03.173 DEBUG:teuthology.orchestra.run.smithi156:osd.4> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T15:54:03.175 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-03-18T15:54:03.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T15:54:03.421 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:03.701 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:03 smithi156 ceph-mon[35919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:03.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:03 smithi150 ceph-mon[31946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:03.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:03.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:03.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:03 smithi191 ceph-mon[36784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:04.684 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-mon[35919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:04.685 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4[47369]: 2024-03-18T15:54:04.612+0000 7efec1f75700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T15:54:04.685 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 15:54:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4[47369]: 2024-03-18T15:54:04.618+0000 7efebdd68700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:54:04.833 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:04.834 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:54:04.834 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:04.834 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:54:04.834 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:04.834 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:04 smithi191 ceph-mon[36784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:04.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:04.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T15:54:04.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: from='osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:04.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T15:54:04.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:04.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:04 smithi150 ceph-mon[31946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0168697 s, 622 MB/s 2024-03-18T15:54:05.387 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-18T15:54:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:05 smithi156 ceph-mon[35919]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:54:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:05 smithi156 ceph-mon[35919]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:54:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:05 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:05 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:05 smithi156 ceph-mon[35919]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:05 smithi191 ceph-mon[36784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:54:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:05 smithi191 ceph-mon[36784]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:54:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:05 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:05 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:05.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:05 smithi191 ceph-mon[36784]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:06.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:05 smithi150 ceph-mon[31946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:54:06.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:05 smithi150 ceph-mon[31946]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T15:54:06.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:05 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:05 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:05 smithi150 ceph-mon[31946]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-18T15:54:06.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-03-18T15:54:06.359 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] boot 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi156 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:06.808 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:06.809 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] boot 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi156 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: osd.4 [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] boot 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi156 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:07.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:07.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:07 smithi191 ceph-mon[36784]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:54:07.827 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:07 smithi191 ceph-mon[36784]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:07 smithi156 ceph-mon[35919]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:54:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:07 smithi156 ceph-mon[35919]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2325919098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0101200 msgr2=0x7fe9c0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2325919098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0101200 0x7fe9c0101620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009990 tx=0x7fe9b0009ca0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2325919098 shutdown_connections 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2325919098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0102da0 0x7fe9c0109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2325919098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0101200 0x7fe9c0101620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2325919098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102400 0x7fe9c0102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2325919098 >> 172.21.15.191:0/2325919098 conn(0x7fe9c00fc7a0 msgr2=0x7fe9c00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.041+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2325919098 shutdown_connections 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2325919098 wait complete. 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 Processor -- start 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 -- start start 2024-03-18T15:54:08.045 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0101200 0x7fe9c019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102da0 0x7fe9c01a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010cde0 con 0x7fe9c0102da0 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010cc60 con 0x7fe9c0101200 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c7b6e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010cf60 con 0x7fe9c0102400 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c5109700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c5109700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:58706/0 (socket says 172.21.15.191:58706) 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c5109700 1 -- 172.21.15.191:0/2431407881 learned_addr learned my addr 172.21.15.191:0/2431407881 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.042+0000 7fe9c590a700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0101200 0x7fe9c019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c610b700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102da0 0x7fe9c01a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 -- 172.21.15.191:0/2431407881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0101200 msgr2=0x7fe9c019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:08.046 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0101200 0x7fe9c019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 -- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102da0 msgr2=0x7fe9c01a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102da0 0x7fe9c01a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 -- 172.21.15.191:0/2431407881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c01a2460 con 0x7fe9c0102400 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c5109700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00d980 tx=0x7fe9bc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc016070 con 0x7fe9c0102400 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c01a2750 con 0x7fe9c0102400 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9bc00de70 con 0x7fe9c0102400 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00f3c0 con 0x7fe9c0102400 2024-03-18T15:54:08.047 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.043+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c01a2c90 con 0x7fe9c0102400 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.044+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c0068f90 con 0x7fe9c0102400 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.045+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe9bc019070 con 0x7fe9c0102400 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.046+0000 7fe9b6ffd700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 0x7fe9ac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.046+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fe9bc0cd570 con 0x7fe9c0102400 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.046+0000 7fe9c590a700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 0x7fe9ac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:08.049 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.046+0000 7fe9c590a700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 0x7fe9ac0a4d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000b5c0 tx=0x7fe9b0003960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:08.051 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.050+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9bc09b680 con 0x7fe9c0102400 2024-03-18T15:54:08.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:07 smithi150 ceph-mon[31946]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T15:54:08.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:07 smithi150 ceph-mon[31946]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:08.254 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:08.253+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe9c00fe9e0 con 0x7fe9ac0a2890 2024-03-18T15:54:08.959 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:08 smithi191 ceph-mon[36784]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:08.959 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:08.959 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:08.959 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:08 smithi156 ceph-mon[35919]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:09.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:08 smithi150 ceph-mon[31946]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:09.838 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:09 smithi191 ceph-mon[36784]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:09 smithi156 ceph-mon[35919]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:10.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:09 smithi150 ceph-mon[31946]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:11 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/3798977279' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:11 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:11 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]': finished 2024-03-18T15:54:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:11 smithi156 ceph-mon[35919]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:54:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:11 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:11.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:11 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/3798977279' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:11 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:11 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]': finished 2024-03-18T15:54:11.292 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:11 smithi191 ceph-mon[36784]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:54:11.293 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:11 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:11.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:11 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/3798977279' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:11 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]: dispatch 2024-03-18T15:54:11.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:11 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53bfde8f-b1e3-4482-ac6a-e17c6a095ea4"}]': finished 2024-03-18T15:54:11.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:11 smithi150 ceph-mon[31946]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T15:54:11.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:11 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:12.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:12 smithi150 ceph-mon[31946]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:12 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/234328799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:12 smithi191 ceph-mon[36784]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:12 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/234328799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:12 smithi156 ceph-mon[35919]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:12 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/234328799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:14.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:14 smithi191 ceph-mon[36784]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:14.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:14 smithi156 ceph-mon[35919]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:14.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:14 smithi150 ceph-mon[31946]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:16.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:16 smithi191 ceph-mon[36784]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:16 smithi156 ceph-mon[35919]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:16.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:16 smithi150 ceph-mon[31946]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:18.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:18 smithi191 ceph-mon[36784]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:18.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:54:18.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:18 smithi156 ceph-mon[35919]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:54:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:18.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:18 smithi150 ceph-mon[31946]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:18.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T15:54:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:19.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:19 smithi191 ceph-mon[36784]: Deploying daemon osd.5 on smithi191 2024-03-18T15:54:19.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:19 smithi156 ceph-mon[35919]: Deploying daemon osd.5 on smithi191 2024-03-18T15:54:19.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:19 smithi150 ceph-mon[31946]: Deploying daemon osd.5 on smithi191 2024-03-18T15:54:20.524 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:20 smithi191 ceph-mon[36784]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:20 smithi156 ceph-mon[35919]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:20.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:20 smithi150 ceph-mon[31946]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:22.533 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:22 smithi191 ceph-mon[36784]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:22 smithi156 ceph-mon[35919]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:22.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:22 smithi150 ceph-mon[31946]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:24 smithi156 ceph-mon[35919]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:24 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:24.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:24 smithi150 ceph-mon[31946]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:24.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:24 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:24.926 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:24 smithi191 ceph-mon[36784]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:24.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:24.927 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:24 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:26.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:26 smithi156 ceph-mon[35919]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:26.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:26 smithi150 ceph-mon[31946]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:26.930 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:26 smithi191 ceph-mon[36784]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:28.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.223+0000 7fe9b6ffd700 1 -- 172.21.15.191:0/2431407881 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe9c00fe9e0 con 0x7fe9ac0a2890 2024-03-18T15:54:28.225 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-03-18T15:54:28.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 msgr2=0x7fe9ac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 0x7fe9ac0a4d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000b5c0 tx=0x7fe9b0003960 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 msgr2=0x7fe9c019ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00d980 tx=0x7fe9bc00dc90 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 shutdown_connections 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c0102da0 0x7fe9c01a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9ac0a2890 0x7fe9ac0a4d50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9c0101200 0x7fe9c019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 --2- 172.21.15.191:0/2431407881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c0102400 0x7fe9c019ad40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 >> 172.21.15.191:0/2431407881 conn(0x7fe9c00fc7a0 msgr2=0x7fe9c00fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 shutdown_connections 2024-03-18T15:54:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:54:28.225+0000 7fe9c7b6e700 1 -- 172.21.15.191:0/2431407881 wait complete. 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:28 smithi191 ceph-mon[36784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:28 smithi156 ceph-mon[35919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.594 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:28 smithi150 ceph-mon[31946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T15:54:28.931 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T15:54:28.934 INFO:tasks.cephadm:Deploying osd.6 on smithi150 with /dev/vg_nvme/lv_2... 2024-03-18T15:54:28.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:54:29.182 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:54:29.305 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.305 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.305 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:54:29.306 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.306 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:54:29.306 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:29.306 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.306 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:29.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:54:29.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:29.880 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T15:54:29.631+0000 7f31e3dd4700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T15:54:29.880 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 15:54:29 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T15:54:29.637+0000 7f31dbbbf700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T15:54:30.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:30.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:30 smithi191 ceph-mon[36784]: from='osd.5 ' entity='osd.5' 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:30.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:30 smithi156 ceph-mon[35919]: from='osd.5 ' entity='osd.5' 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-18T15:54:30.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:30 smithi150 ceph-mon[31946]: from='osd.5 ' entity='osd.5' 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012635 s, 830 MB/s 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 2024-03-18T15:54:31.264 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-18T15:54:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:54:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:54:31.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] boot 2024-03-18T15:54:31.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:54:31.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] boot 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:54:31.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.567 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:54:31.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:54:31.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: osd.5 [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] boot 2024-03-18T15:54:31.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T15:54:31.568 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T15:54:31.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-03-18T15:54:32.209 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:54:32.456 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:32 smithi191 ceph-mon[36784]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:54:32.457 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:32 smithi191 ceph-mon[36784]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:32.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:32 smithi156 ceph-mon[35919]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:54:32.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:32 smithi156 ceph-mon[35919]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:32.540 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:32 smithi150 ceph-mon[31946]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T15:54:32.540 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:32 smithi150 ceph-mon[31946]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi191 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-18T15:54:33.337 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:33.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:33.338 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:33 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi191 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-18T15:54:33.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:33.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:33.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:33 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T15:54:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi191 2024-03-18T15:54:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 8540M 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:33.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:33 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- 172.21.15.150:0/2591728459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 msgr2=0x7f69981028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- 172.21.15.150:0/2591728459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f69981028d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f69880099c0 tx=0x7f6988009cd0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- 172.21.15.150:0/2591728459 shutdown_connections 2024-03-18T15:54:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- 172.21.15.150:0/2591728459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998102e10 0x7f69981096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- 172.21.15.150:0/2591728459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998101270 0x7f6998101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- 172.21.15.150:0/2591728459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f69981028d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- 172.21.15.150:0/2591728459 >> 172.21.15.150:0/2591728459 conn(0x7f69980fc830 msgr2=0x7f69980fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- 172.21.15.150:0/2591728459 shutdown_connections 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- 172.21.15.150:0/2591728459 wait complete. 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 Processor -- start 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- start start 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998101270 0x7f699806b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998102e10 0x7f699819f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.875+0000 7f699f367700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f699810ce40 con 0x7f6998102470 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699f367700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f699810ccc0 con 0x7f6998101270 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699f367700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f699810cfc0 con 0x7f6998102e10 2024-03-18T15:54:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53602/0 (socket says 172.21.15.150:53602) 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 -- 172.21.15.150:0/2084459584 learned_addr learned my addr 172.21.15.150:0/2084459584 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699d103700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998101270 0x7f699806b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699d904700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998102e10 0x7f699819f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 -- 172.21.15.150:0/2084459584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998102e10 msgr2=0x7f699819f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998102e10 0x7f699819f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 -- 172.21.15.150:0/2084459584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998101270 msgr2=0x7f699806b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998101270 0x7f699806b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 -- 172.21.15.150:0/2084459584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69981a01f0 con 0x7f6998102470 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699c902700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f698800fa00 tx=0x7f698800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988027070 con 0x7f6998102470 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69981a0480 con 0x7f6998102470 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6988022950 con 0x7f6998102470 2024-03-18T15:54:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698800bea0 con 0x7f6998102470 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.876+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69981a0930 con 0x7f6998102470 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.877+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6998068f90 con 0x7f6998102470 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.878+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f698802a070 con 0x7f6998102470 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.879+0000 7f698e7fc700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 0x7f69840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.879+0000 7f699d103700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 0x7f69840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.879+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f69880cecc0 con 0x7f6998102470 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.879+0000 7f699d103700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 0x7f69840a4d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69981022d0 tx=0x7f699400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:33.883+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6988013070 con 0x7f6998102470 2024-03-18T15:54:34.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:34.089+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6998062710 con 0x7f69840a2890 2024-03-18T15:54:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:34 smithi156 ceph-mon[35919]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:54:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:34 smithi156 ceph-mon[35919]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:34.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:34 smithi150 ceph-mon[31946]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:54:34.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:34 smithi150 ceph-mon[31946]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:34.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:34.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:34.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:34 smithi191 ceph-mon[36784]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T15:54:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:34 smithi191 ceph-mon[36784]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:54:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:54:34.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:35.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:35 smithi150 ceph-mon[31946]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:35.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:35 smithi150 ceph-mon[31946]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T15:54:35.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:35 smithi156 ceph-mon[35919]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:35.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:35 smithi156 ceph-mon[35919]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T15:54:35.626 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:35 smithi191 ceph-mon[36784]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:54:35.627 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:35 smithi191 ceph-mon[36784]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T15:54:36.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:36 smithi150 ceph-mon[31946]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:54:36.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:36 smithi156 ceph-mon[35919]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:54:36.711 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:36 smithi191 ceph-mon[36784]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:54:37.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:37 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]: dispatch 2024-03-18T15:54:37.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:37 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]': finished 2024-03-18T15:54:37.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:37 smithi156 ceph-mon[35919]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:54:37.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:37.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:37 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]: dispatch 2024-03-18T15:54:37.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:37 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]': finished 2024-03-18T15:54:37.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:37 smithi150 ceph-mon[31946]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:54:37.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:37.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:37 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]: dispatch 2024-03-18T15:54:37.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:37 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2213462879' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7735e727-557c-4104-9d72-5157f53ec694"}]': finished 2024-03-18T15:54:37.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:37 smithi191 ceph-mon[36784]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T15:54:37.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:38.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:38 smithi156 ceph-mon[35919]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:54:38.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:38 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2835821003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:38.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:38 smithi150 ceph-mon[31946]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:54:38.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:38 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2835821003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:38.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:38 smithi191 ceph-mon[36784]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:54:38.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:38 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2835821003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:54:40.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:40 smithi191 ceph-mon[36784]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T15:54:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:40 smithi156 ceph-mon[35919]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T15:54:40.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:40 smithi150 ceph-mon[31946]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-18T15:54:42.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:42 smithi150 ceph-mon[31946]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:54:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:42 smithi191 ceph-mon[36784]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:54:42.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:42 smithi156 ceph-mon[35919]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:54:44.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:44 smithi156 ceph-mon[35919]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:54:44.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:54:44.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:44.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:44 smithi150 ceph-mon[31946]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:54:44.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:54:44.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:44 smithi191 ceph-mon[36784]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:54:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T15:54:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:45.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:45 smithi150 ceph-mon[31946]: Deploying daemon osd.6 on smithi150 2024-03-18T15:54:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:45 smithi156 ceph-mon[35919]: Deploying daemon osd.6 on smithi150 2024-03-18T15:54:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:45 smithi191 ceph-mon[36784]: Deploying daemon osd.6 on smithi150 2024-03-18T15:54:46.584 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:46 smithi150 ceph-mon[31946]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:46.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:46 smithi191 ceph-mon[36784]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:46 smithi156 ceph-mon[35919]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:48 smithi156 ceph-mon[35919]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:48.800 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:48 smithi150 ceph-mon[31946]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:48 smithi191 ceph-mon[36784]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:50.755 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:50 smithi150 ceph-mon[31946]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:50.755 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.755 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.755 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:50 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:50.767 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:50 smithi156 ceph-mon[35919]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:50 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:50 smithi191 ceph-mon[36784]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:50 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:52.707 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:52 smithi150 ceph-mon[31946]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:52 smithi156 ceph-mon[35919]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:52 smithi191 ceph-mon[36784]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:54.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.150+0000 7f698e7fc700 1 -- 172.21.15.150:0/2084459584 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6998062710 con 0x7f69840a2890 2024-03-18T15:54:54.152 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 6 on host 'smithi150' 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 msgr2=0x7f69840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 0x7f69840a4d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69981022d0 tx=0x7f699400b410 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 msgr2=0x7f699819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f698800fa00 tx=0x7f698800fa30 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 shutdown_connections 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6998102e10 0x7f699819f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69840a2890 0x7f69840a4d50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6998101270 0x7f699806b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 --2- 172.21.15.150:0/2084459584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6998102470 0x7f699819d2e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 >> 172.21.15.150:0/2084459584 conn(0x7f69980fc830 msgr2=0x7f6998105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 shutdown_connections 2024-03-18T15:54:54.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:54:54.153+0000 7f699f367700 1 -- 172.21.15.150:0/2084459584 wait complete. 2024-03-18T15:54:54.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:54.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:54.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:54.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T15:54:54.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:54.835 DEBUG:teuthology.orchestra.run.smithi150:osd.6> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T15:54:54.838 INFO:tasks.cephadm:Deploying osd.7 on smithi156 with /dev/vg_nvme/lv_2... 2024-03-18T15:54:54.838 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:54:55.085 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:54:55.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-mon[31946]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:54:55.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-mon[31946]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:54:55.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-mon[31946]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:54:55.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:55.330 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:55.331 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T15:54:55.124+0000 7efe2a8da700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T15:54:55.331 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 15:54:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T15:54:55.134+0000 7efe266cd700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:54:55.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:55 smithi156 ceph-mon[35919]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:54:55.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:55 smithi156 ceph-mon[35919]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:54:55.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:55 smithi156 ceph-mon[35919]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:54:55.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:55.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:55 smithi191 ceph-mon[36784]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T15:54:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:55 smithi191 ceph-mon[36784]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T15:54:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:55 smithi191 ceph-mon[36784]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:54:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:54:56.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:56 smithi150 ceph-mon[31946]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:54:56.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:56 smithi150 ceph-mon[31946]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T15:54:56.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.401 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:56 smithi150 ceph-mon[31946]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:56.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:56 smithi156 ceph-mon[35919]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:54:56.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:56 smithi156 ceph-mon[35919]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T15:54:56.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:56 smithi156 ceph-mon[35919]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:56 smithi191 ceph-mon[36784]: from='osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:54:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:56 smithi191 ceph-mon[36784]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T15:54:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:56 smithi191 ceph-mon[36784]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00846445 s, 1.2 GB/s 2024-03-18T15:54:57.049 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-18T15:54:57.350 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:57 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:54:57.350 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:57 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:54:57.350 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:57 smithi156 ceph-mon[35919]: osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] boot 2024-03-18T15:54:57.350 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:57 smithi156 ceph-mon[35919]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:54:57.351 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:57.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:57 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:54:57.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:57 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:54:57.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:57 smithi150 ceph-mon[31946]: osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] boot 2024-03-18T15:54:57.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:57 smithi150 ceph-mon[31946]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:54:57.408 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:57 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:54:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:57 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:54:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:57 smithi191 ceph-mon[36784]: osd.6 [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] boot 2024-03-18T15:54:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:57 smithi191 ceph-mon[36784]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T15:54:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T15:54:57.703 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-03-18T15:54:57.951 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:54:58.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:58 smithi156 ceph-mon[35919]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T15:54:58.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:58 smithi156 ceph-mon[35919]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:54:58.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:58 smithi191 ceph-mon[36784]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T15:54:58.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:58 smithi191 ceph-mon[36784]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:54:58.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:58 smithi150 ceph-mon[31946]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T15:54:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:58 smithi150 ceph-mon[31946]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:54:59.358 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi150 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 5696M 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:59.359 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:54:59 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2255577852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4103520 msgr2=0x7f4aa4103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:59.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2255577852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4103520 0x7f4aa4103980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4a980099c0 tx=0x7f4a98009cd0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2255577852 shutdown_connections 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2255577852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103ec0 0x7f4aa410a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2255577852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4102320 0x7f4aa4102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2255577852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4103520 0x7f4aa4103980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2255577852 >> 172.21.15.156:0/2255577852 conn(0x7f4aa40fd8e0 msgr2=0x7f4aa40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2255577852 shutdown_connections 2024-03-18T15:54:59.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.570+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2255577852 wait complete. 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 Processor -- start 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 -- start start 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103520 0x7f4aa4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4103ec0 0x7f4aa419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410def0 con 0x7f4aa4103520 2024-03-18T15:54:59.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410dd70 con 0x7f4aa4102320 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa88f1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410e070 con 0x7f4aa4103ec0 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55014/0 (socket says 172.21.15.156:55014) 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.571+0000 7f4aa259c700 1 -- 172.21.15.156:0/2330514391 learned_addr learned my addr 172.21.15.156:0/2330514391 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa2d9d700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4103ec0 0x7f4aa419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa1d9b700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103520 0x7f4aa4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 -- 172.21.15.156:0/2330514391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4103ec0 msgr2=0x7f4aa419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4103ec0 0x7f4aa419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 -- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103520 msgr2=0x7f4aa4198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103520 0x7f4aa4198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 -- 172.21.15.156:0/2330514391 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa41a0140 con 0x7f4aa4102320 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa259c700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400d980 tx=0x7f4a9400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94017030 con 0x7f4aa4102320 2024-03-18T15:54:59.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa41a0430 con 0x7f4aa4102320 2024-03-18T15:54:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a94004500 con 0x7f4aa4102320 2024-03-18T15:54:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94003aa0 con 0x7f4aa4102320 2024-03-18T15:54:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.572+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a09c0 con 0x7f4aa4102320 2024-03-18T15:54:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.573+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa4068f90 con 0x7f4aa4102320 2024-03-18T15:54:59.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.574+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4a9401a070 con 0x7f4aa4102320 2024-03-18T15:54:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.575+0000 7f4a937fe700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 0x7f4a8c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:54:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.575+0000 7f4aa1d9b700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 0x7f4a8c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:54:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.575+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f4a940cd6e0 con 0x7f4aa4102320 2024-03-18T15:54:59.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.576+0000 7f4aa1d9b700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 0x7f4a8c0a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a98000c00 tx=0x7f4a98005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:54:59.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.580+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a940232f0 con 0x7f4aa4102320 2024-03-18T15:54:59.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi150 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 5696M 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:54:59 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi150 2024-03-18T15:54:59.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 5696M 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:54:59.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:54:59 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:54:59.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:54:59.785+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4aa4062710 con 0x7f4a8c0a2890 2024-03-18T15:55:00.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:00 smithi150 ceph-mon[31946]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:00.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:00.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:00.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:00.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:00 smithi156 ceph-mon[35919]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:00.647 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:00.647 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:00.647 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:00.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:00 smithi191 ceph-mon[36784]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:00.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:00.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:00.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:01.565 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:01 smithi156 ceph-mon[35919]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:01.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:01 smithi150 ceph-mon[31946]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:01.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:01 smithi191 ceph-mon[36784]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:02.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:02 smithi150 ceph-mon[31946]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:02.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:02 smithi191 ceph-mon[36784]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:02.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:02 smithi156 ceph-mon[35919]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/2986676388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]': finished 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/2832756967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/2986676388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]': finished 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:03.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/2832756967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/2986676388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]: dispatch 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce254a57-8b1c-4b10-b76e-510130e8d399"}]': finished 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/2832756967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:04 smithi150 ceph-mon[31946]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:04.613 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:04 smithi191 ceph-mon[36784]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:04.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:04 smithi156 ceph-mon[35919]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:06.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:06 smithi156 ceph-mon[35919]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:06.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:06 smithi150 ceph-mon[31946]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:06 smithi191 ceph-mon[36784]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:08.707 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:08 smithi156 ceph-mon[35919]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:08.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:08 smithi150 ceph-mon[31946]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:08 smithi191 ceph-mon[36784]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:10 smithi156 ceph-mon[35919]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:55:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:10 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:10 smithi150 ceph-mon[31946]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:55:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:10 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:10 smithi191 ceph-mon[36784]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T15:55:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:10 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:11.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:11 smithi156 ceph-mon[35919]: Deploying daemon osd.7 on smithi156 2024-03-18T15:55:11.780 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:11 smithi150 ceph-mon[31946]: Deploying daemon osd.7 on smithi156 2024-03-18T15:55:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:11 smithi191 ceph-mon[36784]: Deploying daemon osd.7 on smithi156 2024-03-18T15:55:12.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:12 smithi156 ceph-mon[35919]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:12.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:12 smithi150 ceph-mon[31946]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:12 smithi191 ceph-mon[36784]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:14.581 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:14 smithi156 ceph-mon[35919]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:14.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:14 smithi150 ceph-mon[31946]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:14 smithi191 ceph-mon[36784]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:16.447 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.448 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.448 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:16.448 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:16 smithi156 ceph-mon[35919]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:16.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:16.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:16 smithi191 ceph-mon[36784]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:16.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:16.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:16.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:16 smithi150 ceph-mon[31946]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.151 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 7 on host 'smithi156' 2024-03-18T15:55:19.151 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.149+0000 7f4a937fe700 1 -- 172.21.15.156:0/2330514391 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4aa4062710 con 0x7f4a8c0a2890 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 msgr2=0x7f4a8c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 0x7f4a8c0a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a98000c00 tx=0x7f4a98005fd0 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 msgr2=0x7f4aa41984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400d980 tx=0x7f4a9400dc90 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 shutdown_connections 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4aa4103ec0 0x7f4aa419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4a8c0a2890 0x7f4a8c0a4d50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4102320 0x7f4aa41984e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 --2- 172.21.15.156:0/2330514391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aa4103520 0x7f4aa4198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 >> 172.21.15.156:0/2330514391 conn(0x7f4aa40fd8e0 msgr2=0x7f4aa4106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 shutdown_connections 2024-03-18T15:55:19.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:55:19.152+0000 7f4aa88f1700 1 -- 172.21.15.156:0/2330514391 wait complete. 2024-03-18T15:55:19.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:19 smithi156 ceph-mon[35919]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:19 smithi156 ceph-mon[35919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.766 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:19 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:19.793 DEBUG:teuthology.orchestra.run.smithi156:osd.7> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T15:55:19.796 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_2... 2024-03-18T15:55:19.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T15:55:19.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:19 smithi191 ceph-mon[36784]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:19 smithi191 ceph-mon[36784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.826 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:19 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:19 smithi150 ceph-mon[31946]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:19 smithi150 ceph-mon[31946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T15:55:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:19 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:20.043 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:20.732 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:20 smithi191 ceph-mon[36784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-mon[35919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:20.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T15:55:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: from='osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T15:55:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:20 smithi150 ceph-mon[31946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:55:21.268 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T15:55:20.918+0000 7fa1da4d8700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T15:55:21.268 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 15:55:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T15:55:20.929+0000 7fa1d22c3700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:55:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:21 smithi191 ceph-mon[36784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:55:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:21 smithi191 ceph-mon[36784]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T15:55:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:21 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:21 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.665 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:21 smithi191 ceph-mon[36784]: from='osd.7 ' entity='osd.7' 2024-03-18T15:55:21.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:21 smithi156 ceph-mon[35919]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:55:21.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:21 smithi156 ceph-mon[35919]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T15:55:21.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:21 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:21 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:21 smithi156 ceph-mon[35919]: from='osd.7 ' entity='osd.7' 2024-03-18T15:55:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:21 smithi150 ceph-mon[31946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:55:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:21 smithi150 ceph-mon[31946]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T15:55:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:21 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:21 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:21 smithi150 ceph-mon[31946]: from='osd.7 ' entity='osd.7' 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112978 s, 928 MB/s 2024-03-18T15:55:22.397 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-18T15:55:22.698 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] boot 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:22.699 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:22.700 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] boot 2024-03-18T15:55:22.700 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:55:22.700 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: osd.7 [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] boot 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T15:55:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T15:55:23.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-03-18T15:55:23.410 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:23.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:23.783 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:23.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:23.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:24.580 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:24 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi156 2024-03-18T15:55:24.580 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:24 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-18T15:55:24.580 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:24 smithi191 ceph-mon[36784]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:24.580 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:24 smithi191 ceph-mon[36784]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:55:24.580 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:24 smithi191 ceph-mon[36784]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:55:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:24 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi156 2024-03-18T15:55:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:24 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-18T15:55:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:24 smithi156 ceph-mon[35919]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:24 smithi156 ceph-mon[35919]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:55:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:24 smithi156 ceph-mon[35919]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:55:24.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:24 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi156 2024-03-18T15:55:24.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:24 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-18T15:55:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:24 smithi150 ceph-mon[31946]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:24 smithi150 ceph-mon[31946]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T15:55:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:24 smithi150 ceph-mon[31946]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 -- 172.21.15.191:0/4293080998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08108050 msgr2=0x7fde0810a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 --2- 172.21.15.191:0/4293080998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08108050 0x7fde0810a460 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fde0000b0a0 tx=0x7fde0000b3b0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 -- 172.21.15.191:0/4293080998 shutdown_connections 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 --2- 172.21.15.191:0/4293080998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08108050 0x7fde0810a460 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 --2- 172.21.15.191:0/4293080998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08102df0 0x7fde081051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 --2- 172.21.15.191:0/4293080998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08105720 0x7fde08107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 -- 172.21.15.191:0/4293080998 >> 172.21.15.191:0/4293080998 conn(0x7fde080fc7a0 msgr2=0x7fde080febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 -- 172.21.15.191:0/4293080998 shutdown_connections 2024-03-18T15:55:25.119 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.117+0000 7fde0d566700 1 -- 172.21.15.191:0/4293080998 wait complete. 2024-03-18T15:55:25.121 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.118+0000 7fde0d566700 1 Processor -- start 2024-03-18T15:55:25.121 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.118+0000 7fde0d566700 1 -- start start 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.118+0000 7fde0d566700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.118+0000 7fde0d566700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 0x7fde08198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08108050 0x7fde0819fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810df30 con 0x7fde08108050 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810ddb0 con 0x7fde08105720 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fde0810e0b0 con 0x7fde08102df0 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50352/0 (socket says 172.21.15.191:50352) 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 -- 172.21.15.191:0/2626251904 learned_addr learned my addr 172.21.15.191:0/2626251904 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde067fc700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 0x7fde08198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde077fe700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08108050 0x7fde0819fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:25.122 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 -- 172.21.15.191:0/2626251904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 msgr2=0x7fde08198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 0x7fde08198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 -- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08108050 msgr2=0x7fde0819fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08108050 0x7fde0819fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 -- 172.21.15.191:0/2626251904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde081a01d0 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde067fc700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 0x7fde08198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde06ffd700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fddf800b560 tx=0x7fddf800b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8015020 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a04c0 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf8011d10 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf800f2e0 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.119+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a09b0 con 0x7fde08102df0 2024-03-18T15:55:25.123 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.120+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde08068f90 con 0x7fde08102df0 2024-03-18T15:55:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.121+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fddf8018020 con 0x7fde08102df0 2024-03-18T15:55:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.122+0000 7fddeffff700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 0x7fddf00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.122+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5397+0+0 (secure 0 0 0) 0x7fddf80cdc10 con 0x7fde08102df0 2024-03-18T15:55:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.122+0000 7fde067fc700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 0x7fddf00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.123+0000 7fde067fc700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 0x7fddf00a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005fd0 tx=0x7fddfc005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:55:25.127 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.126+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fddf801f020 con 0x7fde08102df0 2024-03-18T15:55:25.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:25.330+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fde08062710 con 0x7fddf00a2890 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:26.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:26 smithi156 ceph-mon[35919]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:26.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:26 smithi150 ceph-mon[31946]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:26.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:26 smithi191 ceph-mon[36784]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:28.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:28.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/2098985248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]': finished 2024-03-18T15:55:28.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:55:28.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/2098985248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]': finished 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:55:28.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:28.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:28.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/2098985248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]: dispatch 2024-03-18T15:55:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa202baf-1dea-495c-9c8a-637cbbba34c9"}]': finished 2024-03-18T15:55:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T15:55:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:29.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:29 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/3975419830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:29.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:29 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/3975419830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:29 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/3975419830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:30.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:30 smithi150 ceph-mon[31946]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:30.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:30 smithi156 ceph-mon[35919]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:30.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:30 smithi191 ceph-mon[36784]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:32.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:32 smithi191 ceph-mon[36784]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:32 smithi156 ceph-mon[35919]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:32 smithi150 ceph-mon[31946]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:34 smithi191 ceph-mon[36784]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:34.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:34 smithi156 ceph-mon[35919]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:34 smithi150 ceph-mon[31946]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:35.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:35 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:55:35.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:35 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:55:35.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:35 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T15:55:36.700 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:36 smithi191 ceph-mon[36784]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:36.700 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:36.700 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:36 smithi191 ceph-mon[36784]: Deploying daemon osd.8 on smithi191 2024-03-18T15:55:36.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:36 smithi156 ceph-mon[35919]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:36.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:36.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:36 smithi156 ceph-mon[35919]: Deploying daemon osd.8 on smithi191 2024-03-18T15:55:36.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:36 smithi150 ceph-mon[31946]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:36 smithi150 ceph-mon[31946]: Deploying daemon osd.8 on smithi191 2024-03-18T15:55:38.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:38 smithi156 ceph-mon[35919]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:38.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:38 smithi150 ceph-mon[31946]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:38.876 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:38 smithi191 ceph-mon[36784]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:40.693 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:40 smithi191 ceph-mon[36784]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:40.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:40 smithi156 ceph-mon[35919]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:40.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:40 smithi150 ceph-mon[31946]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:42 smithi191 ceph-mon[36784]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:42 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:42.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:42 smithi156 ceph-mon[35919]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:42.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:42 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:42.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:42 smithi150 ceph-mon[31946]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:42 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:44.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:44 smithi191 ceph-mon[36784]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:44.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:44 smithi156 ceph-mon[35919]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:44.841 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:44 smithi150 ceph-mon[31946]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:45.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.300+0000 7fddeffff700 1 -- 172.21.15.191:0/2626251904 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fde08062710 con 0x7fddf00a2890 2024-03-18T15:55:45.303 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 msgr2=0x7fddf00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 0x7fddf00a4d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005fd0 tx=0x7fddfc005f60 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 msgr2=0x7fde08198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fddf800b560 tx=0x7fddf800b920 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 shutdown_connections 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde08108050 0x7fde0819fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde08102df0 0x7fde08198570 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddf00a2890 0x7fddf00a4d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 --2- 172.21.15.191:0/2626251904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde08105720 0x7fde08198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.302+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 >> 172.21.15.191:0/2626251904 conn(0x7fde080fc7a0 msgr2=0x7fde08103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:55:45.305 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.303+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 shutdown_connections 2024-03-18T15:55:45.306 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:55:45.303+0000 7fde0d566700 1 -- 172.21.15.191:0/2626251904 wait complete. 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:45.680 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.681 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:45 smithi191 ceph-mon[36784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:45 smithi156 ceph-mon[35919]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:45 smithi150 ceph-mon[31946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T15:55:45.998 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T15:55:46.000 INFO:tasks.cephadm:Deploying osd.9 on smithi150 with /dev/vg_nvme/lv_1... 2024-03-18T15:55:46.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:55:46.249 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:55:46.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:46.517 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:55:46.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:46.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:55:46.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:46.518 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:46 smithi191 ceph-mon[36784]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:46.624 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:46.624 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:55:46.624 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:46.625 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:55:46.625 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:46.625 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:46 smithi150 ceph-mon[31946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: from='osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:46 smithi156 ceph-mon[35919]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:55:47.962 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 15:55:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T15:55:47.728+0000 7fa31d5fd700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T15:55:47.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 15:55:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T15:55:47.740+0000 7fa3153e8700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121136 s, 866 MB/s 2024-03-18T15:55:48.428 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:48.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:48 smithi150 ceph-mon[31946]: from='osd.8 ' entity='osd.8' 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:48.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:48 smithi191 ceph-mon[36784]: from='osd.8 ' entity='osd.8' 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-03-18T15:55:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:48 smithi156 ceph-mon[35919]: from='osd.8 ' entity='osd.8' 2024-03-18T15:55:49.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-03-18T15:55:49.452 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] boot 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:55:49.574 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:49.575 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] boot 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:55:49.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:49.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:55:49.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:55:49.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: osd.8 [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] boot 2024-03-18T15:55:49.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:55:49.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:55:50.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:50 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi191 2024-03-18T15:55:50.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:50 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-18T15:55:50.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:50 smithi150 ceph-mon[31946]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:50.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:50 smithi150 ceph-mon[31946]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:55:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:50 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi191 2024-03-18T15:55:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:50 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-18T15:55:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:50 smithi191 ceph-mon[36784]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:50 smithi191 ceph-mon[36784]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:55:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:50 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi191 2024-03-18T15:55:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:50 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 5693M 2024-03-18T15:55:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:50 smithi156 ceph-mon[35919]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:50 smithi156 ceph-mon[35919]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T15:55:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 -- 172.21.15.150:0/3797612413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c1056c0 msgr2=0x7ff17c107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 --2- 172.21.15.150:0/3797612413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c1056c0 0x7ff17c107ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff170009990 tx=0x7ff170009ca0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 -- 172.21.15.150:0/3797612413 shutdown_connections 2024-03-18T15:55:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 --2- 172.21.15.150:0/3797612413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c107ff0 0x7ff17c10a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 --2- 172.21.15.150:0/3797612413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 --2- 172.21.15.150:0/3797612413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c1056c0 0x7ff17c107ab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 -- 172.21.15.150:0/3797612413 >> 172.21.15.150:0/3797612413 conn(0x7ff17c0fc7a0 msgr2=0x7ff17c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 -- 172.21.15.150:0/3797612413 shutdown_connections 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.273+0000 7ff1823d1700 1 -- 172.21.15.150:0/3797612413 wait complete. 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 Processor -- start 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 -- start start 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c1056c0 0x7ff17c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10df10 con 0x7ff17c107ff0 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10dd90 con 0x7ff17c1056c0 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff1823d1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10e090 con 0x7ff17c102d90 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36700/0 (socket says 172.21.15.150:36700) 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 -- 172.21.15.150:0/2806334144 learned_addr learned my addr 172.21.15.150:0/2806334144 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff17bfff700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff17b7fe700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c1056c0 0x7ff17c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 -- 172.21.15.150:0/2806334144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 msgr2=0x7ff17c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 -- 172.21.15.150:0/2806334144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c1056c0 msgr2=0x7ff17c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:55:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c1056c0 0x7ff17c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff18096e700 1 -- 172.21.15.150:0/2806334144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c1a01d0 con 0x7ff17c107ff0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.274+0000 7ff17bfff700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff18096e700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff17400add0 tx=0x7ff17400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174017020 con 0x7ff17c107ff0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c1a04c0 con 0x7ff17c107ff0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff174007e30 con 0x7ff17c107ff0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff174005520 con 0x7ff17c107ff0 2024-03-18T15:55:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.275+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c1a0a00 con 0x7ff17c107ff0 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.276+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c068f90 con 0x7ff17c107ff0 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.277+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff174004ad0 con 0x7ff17c107ff0 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.277+0000 7ff1797fa700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 0x7ff1640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.278+0000 7ff17bfff700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 0x7ff1640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.278+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5858+0+0 (secure 0 0 0) 0x7ff1740cd8c0 con 0x7ff17c107ff0 2024-03-18T15:55:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.278+0000 7ff17bfff700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 0x7ff1640a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff16c005950 tx=0x7ff16c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:55:51.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.282+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff174021020 con 0x7ff17c107ff0 2024-03-18T15:55:51.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:55:51.491+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff17c062710 con 0x7ff1640a2890 2024-03-18T15:55:52.355 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:55:52.355 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:52.355 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:52.355 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:52.356 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:52.356 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:52.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:55:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:55:53.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:53 smithi150 ceph-mon[31946]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T15:55:53.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:53 smithi191 ceph-mon[36784]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T15:55:53.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:53 smithi156 ceph-mon[35919]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T15:55:54.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:54 smithi150 ceph-mon[31946]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:55:54.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]: dispatch 2024-03-18T15:55:54.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]': finished 2024-03-18T15:55:54.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:54 smithi150 ceph-mon[31946]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:55:54.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:55:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:54 smithi191 ceph-mon[36784]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:55:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]: dispatch 2024-03-18T15:55:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]': finished 2024-03-18T15:55:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:54 smithi191 ceph-mon[36784]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:55:54.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:55:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:54 smithi156 ceph-mon[35919]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-18T15:55:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]: dispatch 2024-03-18T15:55:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4215755440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e301a14e-5738-44bf-af14-269a3be744dd"}]': finished 2024-03-18T15:55:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:54 smithi156 ceph-mon[35919]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T15:55:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:55:55.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:55 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2966282062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:55.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:55 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2966282062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:55.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:55 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2966282062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:55:56.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:56 smithi150 ceph-mon[31946]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:55:56.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:56 smithi191 ceph-mon[36784]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:55:56.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:56 smithi156 ceph-mon[35919]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:55:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:55:58 smithi150 ceph-mon[31946]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T15:55:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:55:58 smithi156 ceph-mon[35919]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T15:55:58.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:55:58 smithi191 ceph-mon[36784]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T15:56:00.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:00 smithi150 ceph-mon[31946]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:56:00.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:00 smithi156 ceph-mon[35919]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:56:00.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:00 smithi191 ceph-mon[36784]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T15:56:02.735 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:02 smithi150 ceph-mon[31946]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:56:02.735 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:02 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:56:02.735 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:02 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:02.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:02 smithi156 ceph-mon[35919]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:56:02.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:02 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:56:02.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:02 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:02.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:02 smithi191 ceph-mon[36784]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T15:56:02.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:02 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T15:56:02.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:02 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:03.739 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:03 smithi150 ceph-mon[31946]: Deploying daemon osd.9 on smithi150 2024-03-18T15:56:03.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:03 smithi156 ceph-mon[35919]: Deploying daemon osd.9 on smithi150 2024-03-18T15:56:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:03 smithi191 ceph-mon[36784]: Deploying daemon osd.9 on smithi150 2024-03-18T15:56:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:04 smithi150 ceph-mon[31946]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:04.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:04 smithi156 ceph-mon[35919]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:04.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:04 smithi191 ceph-mon[36784]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:06.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:06 smithi156 ceph-mon[35919]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:06.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:06 smithi150 ceph-mon[31946]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:06 smithi191 ceph-mon[36784]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:08.542 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:08 smithi150 ceph-mon[31946]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:08.542 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.542 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.542 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:08 smithi156 ceph-mon[35919]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:08 smithi191 ceph-mon[36784]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:10 smithi156 ceph-mon[35919]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:10.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:10 smithi150 ceph-mon[31946]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:10 smithi191 ceph-mon[36784]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:12.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.008+0000 7ff1797fa700 1 -- 172.21.15.150:0/2806334144 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff17c062710 con 0x7ff1640a2890 2024-03-18T15:56:12.011 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 9 on host 'smithi150' 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.010+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 msgr2=0x7ff1640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.010+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 0x7ff1640a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff16c005950 tx=0x7ff16c0058e0 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.010+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 msgr2=0x7ff17c19fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff17400add0 tx=0x7ff17400c330 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 shutdown_connections 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17c107ff0 0x7ff17c19fb30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff17c102d90 0x7ff17c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff1640a2890 0x7ff1640a4d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 --2- 172.21.15.150:0/2806334144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff17c1056c0 0x7ff17c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 >> 172.21.15.150:0/2806334144 conn(0x7ff17c0fc7a0 msgr2=0x7ff17c1036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:56:12.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 shutdown_connections 2024-03-18T15:56:12.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:56:12.011+0000 7ff1823d1700 1 -- 172.21.15.150:0/2806334144 wait complete. 2024-03-18T15:56:12.341 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:12.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:56:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:12.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:12.734 DEBUG:teuthology.orchestra.run.smithi150:osd.9> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T15:56:12.737 INFO:tasks.cephadm:Deploying osd.10 on smithi156 with /dev/vg_nvme/lv_1... 2024-03-18T15:56:12.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:56:12.985 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:56:13.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-mon[31946]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:56:13.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-mon[31946]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:56:13.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-mon[31946]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:56:13.264 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:13.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:13 smithi156 ceph-mon[35919]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:56:13.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:13 smithi156 ceph-mon[35919]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:56:13.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:13 smithi156 ceph-mon[35919]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:56:13.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:13 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:13.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:13 smithi191 ceph-mon[36784]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T15:56:13.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:13 smithi191 ceph-mon[36784]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T15:56:13.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:13 smithi191 ceph-mon[36784]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-03-18T15:56:13.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:13 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:13.842 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T15:56:13.582+0000 7faa0e5e7700 -1 osd.9 0 waiting for initial osdmap 2024-03-18T15:56:13.842 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 15:56:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T15:56:13.596+0000 7faa05bd1700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:56:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:56:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T15:56:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.179 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:14.180 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:14 smithi150 ceph-mon[31946]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' 2024-03-18T15:56:14.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:56:14.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T15:56:14.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:14.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:14 smithi156 ceph-mon[35919]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-03-18T15:56:14.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:14 smithi191 ceph-mon[36784]: from='osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256]' entity='osd.9' 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00816836 s, 1.3 GB/s 2024-03-18T15:56:14.961 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] boot 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:56:15.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:15 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] boot 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:56:15.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:15 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: osd.9 [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] boot 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T15:56:15.516 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:15 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T15:56:15.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-03-18T15:56:15.832 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:16.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:16.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T15:56:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:16.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:17.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- 172.21.15.156:0/2449226697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 msgr2=0x7ff404101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 --2- 172.21.15.156:0/2449226697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff404101690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40099c0 tx=0x7ff3f4009cd0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- 172.21.15.156:0/2449226697 shutdown_connections 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 --2- 172.21.15.156:0/2449226697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102e10 0x7ff4041096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 --2- 172.21.15.156:0/2449226697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff404101690 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 --2- 172.21.15.156:0/2449226697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102470 0x7ff4041028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- 172.21.15.156:0/2449226697 >> 172.21.15.156:0/2449226697 conn(0x7ff4040fc830 msgr2=0x7ff4040fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- 172.21.15.156:0/2449226697 shutdown_connections 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- 172.21.15.156:0/2449226697 wait complete. 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 Processor -- start 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.375+0000 7ff40b343700 1 -- start start 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102470 0x7ff40419d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102e10 0x7ff40419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410cdf0 con 0x7ff404102e10 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410cc70 con 0x7ff404101270 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410cf70 con 0x7ff404102470 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49682/0 (socket says 172.21.15.156:49682) 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 -- 172.21.15.156:0/2060099960 learned_addr learned my addr 172.21.15.156:0/2060099960 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4088de700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102470 0x7ff40419d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4098e0700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102e10 0x7ff40419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 -- 172.21.15.156:0/2060099960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102470 msgr2=0x7ff40419d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102470 0x7ff40419d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 -- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102e10 msgr2=0x7ff40419f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102e10 0x7ff40419f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 -- 172.21.15.156:0/2060099960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff404197000 con 0x7ff404101270 2024-03-18T15:56:17.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff4090df700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400fa80 tx=0x7ff3f400fab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:56:17.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4027070 con 0x7ff404101270 2024-03-18T15:56:17.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff404197290 con 0x7ff404101270 2024-03-18T15:56:17.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f4022950 con 0x7ff404101270 2024-03-18T15:56:17.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4041977c0 con 0x7ff404101270 2024-03-18T15:56:17.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.376+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4009e60 con 0x7ff404101270 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.377+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff404068f90 con 0x7ff404101270 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.379+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff3f402a070 con 0x7ff404101270 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.380+0000 7ff3fa7fc700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 0x7ff3f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.380+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7ff3f40cefc0 con 0x7ff404101270 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.380+0000 7ff4088de700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 0x7ff3f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.380+0000 7ff4088de700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 0x7ff3f00a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff400009c60 tx=0x7ff400009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:56:17.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.383+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff3f4013070 con 0x7ff404101270 2024-03-18T15:56:17.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:17.589+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff4041081a0 con 0x7ff3f00a2890 2024-03-18T15:56:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:17 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi150 2024-03-18T15:56:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:17 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-18T15:56:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:17 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi150 2024-03-18T15:56:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:17 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-18T15:56:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:17 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi150 2024-03-18T15:56:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:17 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 4272M 2024-03-18T15:56:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:18 smithi156 ceph-mon[35919]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:18 smithi156 ceph-mon[35919]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:18 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:18 smithi191 ceph-mon[36784]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:18 smithi191 ceph-mon[36784]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:18 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:18 smithi150 ceph-mon[31946]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:18 smithi150 ceph-mon[31946]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:18 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:19.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:19 smithi156 ceph-mon[35919]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:19 smithi191 ceph-mon[36784]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:19 smithi150 ceph-mon[31946]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:20 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/329426228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:20 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:20 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]': finished 2024-03-18T15:56:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:20 smithi191 ceph-mon[36784]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:56:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:20 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/329426228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:20 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:20 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]': finished 2024-03-18T15:56:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:20 smithi156 ceph-mon[35919]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:56:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:21.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:20 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/329426228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:21.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:20 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]: dispatch 2024-03-18T15:56:21.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:20 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f122516-01f0-41e8-9ff8-c73bbf6609d1"}]': finished 2024-03-18T15:56:21.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:20 smithi150 ceph-mon[31946]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T15:56:21.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:21 smithi191 ceph-mon[36784]: from='client.? 172.21.15.156:0/2222891031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:21 smithi191 ceph-mon[36784]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:21 smithi156 ceph-mon[35919]: from='client.? 172.21.15.156:0/2222891031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:21 smithi156 ceph-mon[35919]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:21 smithi150 ceph-mon[31946]: from='client.? 172.21.15.156:0/2222891031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:21 smithi150 ceph-mon[31946]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:24.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:24 smithi156 ceph-mon[35919]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:24 smithi150 ceph-mon[31946]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:24 smithi191 ceph-mon[36784]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:26.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:26 smithi156 ceph-mon[35919]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:26.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:26 smithi150 ceph-mon[31946]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:26.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:26 smithi191 ceph-mon[36784]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:28.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:28 smithi156 ceph-mon[35919]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:28.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:56:28.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:28 smithi150 ceph-mon[31946]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:56:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:28 smithi191 ceph-mon[36784]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T15:56:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:29.735 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:29 smithi156 ceph-mon[35919]: Deploying daemon osd.10 on smithi156 2024-03-18T15:56:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:29 smithi150 ceph-mon[31946]: Deploying daemon osd.10 on smithi156 2024-03-18T15:56:30.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:29 smithi191 ceph-mon[36784]: Deploying daemon osd.10 on smithi156 2024-03-18T15:56:30.737 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:30 smithi156 ceph-mon[35919]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:30.743 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:30 smithi150 ceph-mon[31946]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:30.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:30 smithi191 ceph-mon[36784]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:32.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:32 smithi156 ceph-mon[35919]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:32 smithi150 ceph-mon[31946]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:32.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:32 smithi191 ceph-mon[36784]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:34 smithi156 ceph-mon[35919]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:34 smithi191 ceph-mon[36784]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:34 smithi150 ceph-mon[31946]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:36.696 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:36 smithi156 ceph-mon[35919]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:36.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:36 smithi191 ceph-mon[36784]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:36 smithi150 ceph-mon[31946]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:37.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.408+0000 7ff3fa7fc700 1 -- 172.21.15.156:0/2060099960 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff4041081a0 con 0x7ff3f00a2890 2024-03-18T15:56:37.409 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 10 on host 'smithi156' 2024-03-18T15:56:37.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 msgr2=0x7ff3f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:37.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 0x7ff3f00a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff400009c60 tx=0x7ff400009260 comp rx=0 tx=0).stop 2024-03-18T15:56:37.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 msgr2=0x7ff40406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:37.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400fa80 tx=0x7ff3f400fab0 comp rx=0 tx=0).stop 2024-03-18T15:56:37.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 shutdown_connections 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff404102e10 0x7ff40419f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff3f00a2890 0x7ff3f00a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff404101270 0x7ff40406b590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 --2- 172.21.15.156:0/2060099960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff404102470 0x7ff40419d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 >> 172.21.15.156:0/2060099960 conn(0x7ff4040fc830 msgr2=0x7ff4041057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 shutdown_connections 2024-03-18T15:56:37.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-18T15:56:37.410+0000 7ff40b343700 1 -- 172.21.15.156:0/2060099960 wait complete. 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:37.702 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:37.703 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.703 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.703 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:37 smithi156 ceph-mon[35919]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:37 smithi191 ceph-mon[36784]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:38.012 DEBUG:teuthology.orchestra.run.smithi156:osd.10> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T15:56:38.014 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2024-03-18T15:56:38.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T15:56:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:56:38.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:38.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:38.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:37 smithi150 ceph-mon[31946]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:38.261 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:56:38.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:38 smithi191 ceph-mon[36784]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:56:38.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:38 smithi191 ceph-mon[36784]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:38.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:38 smithi191 ceph-mon[36784]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:56:38.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:38 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:38.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:38 smithi191 ceph-mon[36784]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-mon[35919]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:56:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-mon[35919]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-mon[35919]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:56:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-mon[35919]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:39.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T15:56:38.650+0000 7fd04d7b1700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T15:56:39.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 15:56:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T15:56:38.662+0000 7fd0495a4700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:56:39.091 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:38 smithi150 ceph-mon[31946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T15:56:39.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:38 smithi150 ceph-mon[31946]: from='osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:39.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:38 smithi150 ceph-mon[31946]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T15:56:39.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:38 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:38 smithi150 ceph-mon[31946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-18T15:56:39.695 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:56:39.695 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T15:56:39.695 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.695 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.695 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] boot 2024-03-18T15:56:39.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:56:39.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.696 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:39 smithi191 ceph-mon[36784]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] boot 2024-03-18T15:56:39.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:56:39.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:39.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:39 smithi156 ceph-mon[35919]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: osd.10 [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] boot 2024-03-18T15:56:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T15:56:40.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T15:56:40.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:39 smithi150 ceph-mon[31946]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 312 MiB used, 894 GiB / 894 GiB avail 2024-03-18T15:56:40.420 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T15:56:40.420 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T15:56:40.420 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-03-18T15:56:40.420 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-03-18T15:56:40.420 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113454 s, 924 MB/s 2024-03-18T15:56:40.421 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 2024-03-18T15:56:40.421 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-03-18T15:56:41.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-03-18T15:56:41.452 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:41.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:41 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:41.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:41.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:56:41.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:41.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:41.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:56:41.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:41 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:56:42.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:42 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi156 2024-03-18T15:56:42.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:42 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-18T15:56:42.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:42 smithi156 ceph-mon[35919]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:56:42.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:42 smithi156 ceph-mon[35919]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:42.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:42 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi156 2024-03-18T15:56:42.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:42 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-18T15:56:42.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:42 smithi150 ceph-mon[31946]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:56:42.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:42 smithi150 ceph-mon[31946]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:42.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:42 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi156 2024-03-18T15:56:42.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:42 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-18T15:56:42.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:42 smithi191 ceph-mon[36784]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T15:56:42.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:42 smithi191 ceph-mon[36784]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1619615151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103500 msgr2=0x7f2e4c103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1619615151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103500 0x7f2e4c103960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009990 tx=0x7f2e38009ca0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1619615151 shutdown_connections 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1619615151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1619615151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c102300 0x7f2e4c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1619615151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103500 0x7f2e4c103960 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1619615151 >> 172.21.15.191:0/1619615151 conn(0x7f2e4c0fd8e0 msgr2=0x7f2e4c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1619615151 shutdown_connections 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1619615151 wait complete. 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 Processor -- start 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.280+0000 7f2e52c3a700 1 -- start start 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 0x7f2e4c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c103500 0x7f2e4c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c10ded0 con 0x7f2e4c102300 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c10dd50 con 0x7f2e4c103500 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c10e050 con 0x7f2e4c103ea0 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:55498/0 (socket says 172.21.15.191:55498) 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 -- 172.21.15.191:0/1056780462 learned_addr learned my addr 172.21.15.191:0/1056780462 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e509d6700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 0x7f2e4c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e4bfff700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c103500 0x7f2e4c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 -- 172.21.15.191:0/1056780462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c103500 msgr2=0x7f2e4c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c103500 0x7f2e4c1989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 -- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 msgr2=0x7f2e4c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 0x7f2e4c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 -- 172.21.15.191:0/1056780462 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4c1a00e0 con 0x7f2e4c103ea0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e509d6700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 0x7f2e4c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e511d7700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400add0 tx=0x7f2e4400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44017020 con 0x7f2e4c103ea0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e4400ce20 con 0x7f2e4c103ea0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4c1a03d0 con 0x7f2e4c103ea0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.281+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44005520 con 0x7f2e4c103ea0 2024-03-18T15:56:43.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.282+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4c1a08c0 con 0x7f2e4c103ea0 2024-03-18T15:56:43.286 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.283+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e4c068f90 con 0x7f2e4c103ea0 2024-03-18T15:56:43.286 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:43 smithi191 ceph-mon[36784]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:56:43.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.284+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2e44004ad0 con 0x7f2e4c103ea0 2024-03-18T15:56:43.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.284+0000 7f2e49ffb700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 0x7f2e3c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:56:43.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.285+0000 7f2e509d6700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 0x7f2e3c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:56:43.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.285+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6694+0+0 (secure 0 0 0) 0x7f2e440cdc10 con 0x7f2e4c103ea0 2024-03-18T15:56:43.287 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.285+0000 7f2e509d6700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 0x7f2e3c0a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c103360 tx=0x7f2e4000a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:56:43.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.289+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e44021020 con 0x7f2e4c103ea0 2024-03-18T15:56:43.495 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:56:43.494+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2e4c062710 con 0x7f2e3c0a2890 2024-03-18T15:56:43.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:43 smithi150 ceph-mon[31946]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:56:43.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:43 smithi156 ceph-mon[35919]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T15:56:44.353 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T15:56:44.354 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:56:44.354 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:44.354 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:44.354 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:44.354 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:44 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:44.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:44 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T15:56:44.670 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:44 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:56:46.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/1319560945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]': finished 2024-03-18T15:56:46.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:56:46.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/1319560945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]': finished 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:56:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:56:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T15:56:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/1319560945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]: dispatch 2024-03-18T15:56:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0691d334-c620-436b-b0f8-8343b4c4cf33"}]': finished 2024-03-18T15:56:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T15:56:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:56:47.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.191:0/2195253461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:47.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.191:0/2195253461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.191:0/2195253461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T15:56:48.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:48 smithi156 ceph-mon[35919]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:56:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:48 smithi150 ceph-mon[31946]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:56:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:48 smithi191 ceph-mon[36784]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T15:56:50.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:50 smithi191 ceph-mon[36784]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:56:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:50 smithi156 ceph-mon[35919]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:56:50.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:50 smithi150 ceph-mon[31946]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T15:56:52.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:52 smithi156 ceph-mon[35919]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:52.790 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:52 smithi191 ceph-mon[36784]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:52 smithi150 ceph-mon[31946]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:54.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:54 smithi156 ceph-mon[35919]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:56:54.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:54.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:54 smithi191 ceph-mon[36784]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:54.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:56:54.769 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:54 smithi150 ceph-mon[31946]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T15:56:54.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:56:55.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:55 smithi156 ceph-mon[35919]: Deploying daemon osd.11 on smithi191 2024-03-18T15:56:55.788 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:55 smithi191 ceph-mon[36784]: Deploying daemon osd.11 on smithi191 2024-03-18T15:56:55.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:55 smithi150 ceph-mon[31946]: Deploying daemon osd.11 on smithi191 2024-03-18T15:56:56.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:56 smithi156 ceph-mon[35919]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:56 smithi150 ceph-mon[31946]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:56.852 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:56 smithi191 ceph-mon[36784]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:58.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:56:58 smithi156 ceph-mon[35919]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:56:58 smithi150 ceph-mon[31946]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:56:58.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:56:58 smithi191 ceph-mon[36784]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:00.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:00 smithi156 ceph-mon[35919]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:00.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:00 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:00.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:00 smithi191 ceph-mon[36784]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:00.772 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.773 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:00 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:00.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:00 smithi150 ceph-mon[31946]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:00.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:00.871 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:00 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:02.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:02 smithi156 ceph-mon[35919]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:02.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:02 smithi150 ceph-mon[31946]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:02.862 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:02 smithi191 ceph-mon[36784]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:04.165 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.164+0000 7f2e49ffb700 1 -- 172.21.15.191:0/1056780462 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2e4c062710 con 0x7f2e3c0a2890 2024-03-18T15:57:04.166 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 msgr2=0x7f2e3c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 0x7f2e3c0a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c103360 tx=0x7f2e4000a680 comp rx=0 tx=0).stop 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 msgr2=0x7f2e4c19fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400add0 tx=0x7f2e4400c330 comp rx=0 tx=0).stop 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 shutdown_connections 2024-03-18T15:57:04.168 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2e4c103ea0 0x7f2e4c19fa40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e4c102300 0x7f2e4c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2e3c0a2890 0x7f2e3c0a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 --2- 172.21.15.191:0/1056780462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c103500 0x7f2e4c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 >> 172.21.15.191:0/1056780462 conn(0x7f2e4c0fd8e0 msgr2=0x7f2e4c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 shutdown_connections 2024-03-18T15:57:04.169 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:04.167+0000 7f2e52c3a700 1 -- 172.21.15.191:0/1056780462 wait complete. 2024-03-18T15:57:04.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:04.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:04 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:04 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T15:57:04.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:04.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:04 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:04.822 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T15:57:04.824 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T15:57:04.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T15:57:05.078 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:57:05.367 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-mon[36784]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:57:05.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-mon[36784]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:57:05.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:05.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-mon[36784]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:05.368 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-mon[36784]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:05.368 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T15:57:05.233+0000 7fc366c03700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T15:57:05.368 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 15:57:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T15:57:05.253+0000 7fc3629f6700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T15:57:05.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:05 smithi156 ceph-mon[35919]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:57:05.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:05 smithi156 ceph-mon[35919]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:57:05.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:05 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:05.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:05 smithi156 ceph-mon[35919]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:05.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:05 smithi156 ceph-mon[35919]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:05.525 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:05 smithi150 ceph-mon[31946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T15:57:05.525 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:05 smithi150 ceph-mon[31946]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T15:57:05.525 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:05 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:05.526 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:05 smithi150 ceph-mon[31946]: from='osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:05.526 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:05 smithi150 ceph-mon[31946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-03-18T15:57:06.380 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:06 smithi191 ceph-mon[36784]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:57:06.380 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:06 smithi191 ceph-mon[36784]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T15:57:06.380 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.380 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:06 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.380 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:06 smithi191 ceph-mon[36784]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:06.464 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:06 smithi150 ceph-mon[31946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:57:06.464 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:06 smithi150 ceph-mon[31946]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T15:57:06.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:06 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.465 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:06 smithi150 ceph-mon[31946]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:06.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:06 smithi156 ceph-mon[35919]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-03-18T15:57:06.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:06 smithi156 ceph-mon[35919]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T15:57:06.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:06 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:06.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:06 smithi156 ceph-mon[35919]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- 172.21.15.150:0/913871516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 msgr2=0x7ff848102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 --2- 172.21.15.150:0/913871516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848102720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff8380099c0 tx=0x7ff838009cd0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- 172.21.15.150:0/913871516 shutdown_connections 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 --2- 172.21.15.150:0/913871516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103ea0 0x7ff84810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 --2- 172.21.15.150:0/913871516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848102720 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 --2- 172.21.15.150:0/913871516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103500 0x7ff848103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- 172.21.15.150:0/913871516 >> 172.21.15.150:0/913871516 conn(0x7ff8480fd8e0 msgr2=0x7ff8480ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- 172.21.15.150:0/913871516 shutdown_connections 2024-03-18T15:57:06.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- 172.21.15.150:0/913871516 wait complete. 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 Processor -- start 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.777+0000 7ff84e11f700 1 -- start start 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 0x7ff848198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103ea0 0x7ff84819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff84810dec0 con 0x7ff848102300 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff84810dd40 con 0x7ff848103500 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff84810e040 con 0x7ff848103ea0 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41988/0 (socket says 172.21.15.150:41988) 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 -- 172.21.15.150:0/1758303763 learned_addr learned my addr 172.21.15.150:0/1758303763 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff847fff700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103ea0 0x7ff84819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff846ffd700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 0x7ff848198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:06.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 -- 172.21.15.150:0/1758303763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103ea0 msgr2=0x7ff84819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103ea0 0x7ff84819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 -- 172.21.15.150:0/1758303763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 msgr2=0x7ff848198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 0x7ff848198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 -- 172.21.15.150:0/1758303763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8481a0190 con 0x7ff848102300 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff846ffd700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 0x7ff848198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff8477fe700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff83800fab0 tx=0x7ff83800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff838027070 con 0x7ff848102300 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8481a0420 con 0x7ff848102300 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff83800fdb0 con 0x7ff848102300 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.778+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8380173f0 con 0x7ff848102300 2024-03-18T15:57:06.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.779+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8481a0950 con 0x7ff848102300 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.779+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff848068f90 con 0x7ff848102300 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.781+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff83802a070 con 0x7ff848102300 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.782+0000 7ff844ff9700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 0x7ff8300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.782+0000 7ff846ffd700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 0x7ff8300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.782+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff8380cf320 con 0x7ff848102300 2024-03-18T15:57:06.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.782+0000 7ff846ffd700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 0x7ff8300a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009be0 tx=0x7ff83c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:06.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.785+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff838013070 con 0x7ff848102300 2024-03-18T15:57:06.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.981+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff848062ee0 con 0x7ff848102300 2024-03-18T15:57:06.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.982+0000 7ff844ff9700 1 -- 172.21.15.150:0/1758303763 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7ff83809c8f0 con 0x7ff848102300 2024-03-18T15:57:06.983 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:06.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 msgr2=0x7ff8300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:06.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 0x7ff8300a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009be0 tx=0x7ff83c009260 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 msgr2=0x7ff848198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff83800fab0 tx=0x7ff83800fae0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 shutdown_connections 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff848103ea0 0x7ff84819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff848102300 0x7ff848198530 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff8300a2890 0x7ff8300a4d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 --2- 172.21.15.150:0/1758303763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff848103500 0x7ff848198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 >> 172.21.15.150:0/1758303763 conn(0x7ff8480fd8e0 msgr2=0x7ff848106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 shutdown_connections 2024-03-18T15:57:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:06.984+0000 7ff84e11f700 1 -- 172.21.15.150:0/1758303763 wait complete. 2024-03-18T15:57:07.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: purged_snaps scrub starts 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: purged_snaps scrub ok 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] boot 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:07 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1758303763' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: purged_snaps scrub starts 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: purged_snaps scrub ok 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] boot 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:07 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1758303763' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: purged_snaps scrub starts 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: purged_snaps scrub ok 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: osd.11 [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] boot 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T15:57:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:07 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1758303763' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T15:57:07.685 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710777426,"num_in_osds":12,"osd_in_since":1710777406,"num_remapped_pgs":0} 2024-03-18T15:57:07.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:57:07.942 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:57:08.327 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:08.328 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:08 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:57:08.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:08.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:08.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:08 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:08.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:08 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:09.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:09 smithi150 ceph-mon[31946]: Detected new or changed devices on smithi191 2024-03-18T15:57:09.495 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:09 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-18T15:57:09.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:09 smithi156 ceph-mon[35919]: Detected new or changed devices on smithi191 2024-03-18T15:57:09.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:09 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-18T15:57:09.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:09 smithi191 ceph-mon[36784]: Detected new or changed devices on smithi191 2024-03-18T15:57:09.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:09 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 4270M 2024-03-18T15:57:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3180011832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8106160 msgr2=0x7fc8e8106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3180011832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8106160 0x7fc8e8106540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009990 tx=0x7fc8d8009ca0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3180011832 shutdown_connections 2024-03-18T15:57:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3180011832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8101ee0 0x7fc8e810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3180011832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8106160 0x7fc8e8106540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3180011832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101520 0x7fc8e81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3180011832 >> 172.21.15.150:0/3180011832 conn(0x7fc8e8077f80 msgr2=0x7fc8e8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:09.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3180011832 shutdown_connections 2024-03-18T15:57:09.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.748+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3180011832 wait complete. 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 Processor -- start 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 -- start start 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101ee0 0x7fc8e8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 0x7fc8e81a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e8111540 con 0x7fc8e8101520 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e81113c0 con 0x7fc8e8101ee0 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8ecfdd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e81116c0 con 0x7fc8e8106160 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42016/0 (socket says 172.21.15.150:42016) 2024-03-18T15:57:09.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 -- 172.21.15.150:0/3464193106 learned_addr learned my addr 172.21.15.150:0/3464193106 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e6d9d700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 0x7fc8e81a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e5d9b700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101ee0 0x7fc8e8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 -- 172.21.15.150:0/3464193106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 msgr2=0x7fc8e81a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 0x7fc8e81a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 -- 172.21.15.150:0/3464193106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101ee0 msgr2=0x7fc8e8199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101ee0 0x7fc8e8199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e659c700 1 -- 172.21.15.150:0/3464193106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e81a4d80 con 0x7fc8e8101520 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.749+0000 7fc8e6d9d700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 0x7fc8e81a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8e659c700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8000c00 tx=0x7fc8d8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8027070 con 0x7fc8e8101520 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e81a4f50 con 0x7fc8e8101520 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8d8004190 con 0x7fc8e8101520 2024-03-18T15:57:09.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d800f460 con 0x7fc8e8101520 2024-03-18T15:57:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.750+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e81a5340 con 0x7fc8e8101520 2024-03-18T15:57:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.751+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e8068f90 con 0x7fc8e8101520 2024-03-18T15:57:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.752+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc8d802a070 con 0x7fc8e8101520 2024-03-18T15:57:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.752+0000 7fc8d77fe700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 0x7fc8d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.753+0000 7fc8e5d9b700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 0x7fc8d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:09.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.753+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc8d80cf8a0 con 0x7fc8e8101520 2024-03-18T15:57:09.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.753+0000 7fc8e5d9b700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 0x7fc8d00a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8e81013f0 tx=0x7fc8dc005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:09.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.756+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8d8013070 con 0x7fc8e8101520 2024-03-18T15:57:09.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.951+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8e81a55d0 con 0x7fc8e8101520 2024-03-18T15:57:09.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.952+0000 7fc8d77fe700 1 -- 172.21.15.150:0/3464193106 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18512 (secure 0 0 0) 0x7fc8d809ce70 con 0x7fc8e8101520 2024-03-18T15:57:09.954 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:09.954 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":82,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:07.237328+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:57:09.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 msgr2=0x7fc8d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:09.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 0x7fc8d00a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8e81013f0 tx=0x7fc8dc005d50 comp rx=0 tx=0).stop 2024-03-18T15:57:09.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 msgr2=0x7fc8e81997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8000c00 tx=0x7fc8d8003730 comp rx=0 tx=0).stop 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 shutdown_connections 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8e8106160 0x7fc8e81a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8e8101520 0x7fc8e81997c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc8d00a2890 0x7fc8d00a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 --2- 172.21.15.150:0/3464193106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc8e8101ee0 0x7fc8e8199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 >> 172.21.15.150:0/3464193106 conn(0x7fc8e8077f80 msgr2=0x7fc8e80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 shutdown_connections 2024-03-18T15:57:09.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:09.955+0000 7fc8ecfdd700 1 -- 172.21.15.150:0/3464193106 wait complete. 2024-03-18T15:57:10.378 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:10 smithi150 ceph-mon[31946]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:10.378 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:10 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3464193106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:10.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:10 smithi156 ceph-mon[35919]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:10.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:10 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3464193106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:10.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:10 smithi191 ceph-mon[36784]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:10.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:10 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3464193106' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:10.779 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:57:10.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T15:57:11.018 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 -- 172.21.15.150:0/3893863960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc101490 msgr2=0x7efcdc101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 --2- 172.21.15.150:0/3893863960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc101490 0x7efcdc101870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efcd00099c0 tx=0x7efcd0009cd0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 -- 172.21.15.150:0/3893863960 shutdown_connections 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 --2- 172.21.15.150:0/3893863960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcdc1064e0 0x7efcdc10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 --2- 172.21.15.150:0/3893863960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc101490 0x7efcdc101870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 --2- 172.21.15.150:0/3893863960 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101e40 0x7efcdc105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.675+0000 7efce2e85700 1 -- 172.21.15.150:0/3893863960 >> 172.21.15.150:0/3893863960 conn(0x7efcdc077220 msgr2=0x7efcdc077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- 172.21.15.150:0/3893863960 shutdown_connections 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- 172.21.15.150:0/3893863960 wait complete. 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 Processor -- start 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- start start 2024-03-18T15:57:12.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101490 0x7efcdc1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcdc101e40 0x7efcdc199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc1114e0 con 0x7efcdc1064e0 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc111360 con 0x7efcdc101e40 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.676+0000 7efce2e85700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc111660 con 0x7efcdc101490 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59104/0 (socket says 172.21.15.150:59104) 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 -- 172.21.15.150:0/652262934 learned_addr learned my addr 172.21.15.150:0/652262934 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce0c21700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101490 0x7efcdc1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 -- 172.21.15.150:0/652262934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101490 msgr2=0x7efcdc1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101490 0x7efcdc1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 -- 172.21.15.150:0/652262934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcdc101e40 msgr2=0x7efcdc199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:12.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcdc101e40 0x7efcdc199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 -- 172.21.15.150:0/652262934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc1a4ea0 con 0x7efcdc1064e0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce1422700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efcd4007ee0 tx=0x7efcd400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd400ddd0 con 0x7efcdc1064e0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcdc1a50d0 con 0x7efcdc1064e0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcd4004020 con 0x7efcdc1064e0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4015660 con 0x7efcdc1064e0 2024-03-18T15:57:12.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.677+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcdc1a5610 con 0x7efcdc1064e0 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.678+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcdc068f90 con 0x7efcdc1064e0 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.679+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efcd40041d0 con 0x7efcdc1064e0 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.680+0000 7efcd9ffb700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 0x7efccc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.680+0000 7efce0c21700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 0x7efccc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.680+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efcd40ce7b0 con 0x7efcdc1064e0 2024-03-18T15:57:12.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.680+0000 7efce0c21700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 0x7efccc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efcd0000c00 tx=0x7efcd0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:12.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.684+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcd4010070 con 0x7efcdc1064e0 2024-03-18T15:57:12.685 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:12 smithi150 ceph-mon[31946]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:12.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:12 smithi156 ceph-mon[35919]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:12.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.876+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcdc02dd30 con 0x7efcdc1064e0 2024-03-18T15:57:12.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.876+0000 7efcd9ffb700 1 -- 172.21.15.150:0/652262934 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7efcd409bcf0 con 0x7efcdc1064e0 2024-03-18T15:57:12.878 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 msgr2=0x7efccc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 0x7efccc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efcd0000c00 tx=0x7efcd0005fb0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 msgr2=0x7efcdc1a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efcd4007ee0 tx=0x7efcd400d3b0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 shutdown_connections 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc1064e0 0x7efcdc1a4960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efcdc101490 0x7efcdc1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7efccc0a2890 0x7efccc0a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 --2- 172.21.15.150:0/652262934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcdc101e40 0x7efcdc199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 >> 172.21.15.150:0/652262934 conn(0x7efcdc077220 msgr2=0x7efcdc105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:12.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 shutdown_connections 2024-03-18T15:57:12.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:12.879+0000 7efce2e85700 1 -- 172.21.15.150:0/652262934 wait complete. 2024-03-18T15:57:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:12 smithi191 ceph-mon[36784]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:13.489 INFO:tasks.cephadm:Adding mds.a on smithi150 2024-03-18T15:57:13.489 INFO:tasks.cephadm:Adding mds.d on smithi150 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.g on smithi150 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.j on smithi150 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.b on smithi156 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.e on smithi156 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.h on smithi156 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.k on smithi156 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.c on smithi191 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.f on smithi191 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.i on smithi191 2024-03-18T15:57:13.490 INFO:tasks.cephadm:Adding mds.l on smithi191 2024-03-18T15:57:13.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l' 2024-03-18T15:57:13.739 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:57:13.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:13 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/652262934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:13 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/652262934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:13.861 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:13 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/652262934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:14 smithi156 ceph-mon[35919]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:14.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:14 smithi150 ceph-mon[31946]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:14.885 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:14 smithi191 ceph-mon[36784]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 -- 172.21.15.191:0/1409637110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 msgr2=0x7feb68103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/1409637110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb68103d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb5c009990 tx=0x7feb5c009ca0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 -- 172.21.15.191:0/1409637110 shutdown_connections 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/1409637110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 0x7feb6810a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/1409637110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb68103d10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/1409637110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb681007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 -- 172.21.15.191:0/1409637110 >> 172.21.15.191:0/1409637110 conn(0x7feb680fbf10 msgr2=0x7feb680fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.443+0000 7feb6f0cc700 1 -- 172.21.15.191:0/1409637110 shutdown_connections 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 -- 172.21.15.191:0/1409637110 wait complete. 2024-03-18T15:57:15.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 Processor -- start 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 -- start start 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 0x7feb68198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb6819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb6810df50 con 0x7feb68100370 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb6810ddd0 con 0x7feb68100d30 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.444+0000 7feb6f0cc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7feb6810e0d0 con 0x7feb681038f0 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6ce68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb6819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6ce68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.191:36858/0 (socket says 172.21.15.191:36858) 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6ce68700 1 -- 172.21.15.191:0/3045659710 learned_addr learned my addr 172.21.15.191:0/3045659710 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb67fff700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 0x7feb68198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 -- 172.21.15.191:0/3045659710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 msgr2=0x7feb68198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 0x7feb68198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 -- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 msgr2=0x7feb68198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 -- 172.21.15.191:0/3045659710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb681a01a0 con 0x7feb681038f0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6ce68700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6d669700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb6819fb00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6000be50 tx=0x7feb60009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60019070 con 0x7feb681038f0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb681a0490 con 0x7feb681038f0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6000cb90 con 0x7feb681038f0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.445+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600125a0 con 0x7feb681038f0 2024-03-18T15:57:15.449 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.446+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb681a0a50 con 0x7feb681038f0 2024-03-18T15:57:15.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.446+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb68068f90 con 0x7feb681038f0 2024-03-18T15:57:15.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.447+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7feb6001c070 con 0x7feb681038f0 2024-03-18T15:57:15.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.448+0000 7feb65ffb700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 0x7feb580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:15.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.448+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7feb600cda60 con 0x7feb681038f0 2024-03-18T15:57:15.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.448+0000 7feb6ce68700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 0x7feb580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:15.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.449+0000 7feb6ce68700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 0x7feb580a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00b5c0 tx=0x7feb5c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:15.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.453+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb6009afa0 con 0x7feb681038f0 2024-03-18T15:57:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.659+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}) -- 0x7feb68062910 con 0x7feb580a2890 2024-03-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.669+0000 7feb65ffb700 1 -- 172.21.15.191:0/3045659710 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7feb68062910 con 0x7feb580a2890 2024-03-18T15:57:15.670 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mds.all update... 2024-03-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 msgr2=0x7feb580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 0x7feb580a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00b5c0 tx=0x7feb5c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 msgr2=0x7feb6819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb6819fb00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6000be50 tx=0x7feb60009ed0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 shutdown_connections 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb681038f0 0x7feb6819fb00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb68100370 0x7feb68198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7feb580a2890 0x7feb580a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 --2- 172.21.15.191:0/3045659710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb68100d30 0x7feb68198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 >> 172.21.15.191:0/3045659710 conn(0x7feb680fbf10 msgr2=0x7feb68106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 shutdown_connections 2024-03-18T15:57:15.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-18T15:57:15.671+0000 7feb6f0cc700 1 -- 172.21.15.191:0/3045659710 wait complete. 2024-03-18T15:57:16.359 DEBUG:teuthology.orchestra.run.smithi150:mds.a> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T15:57:16.361 DEBUG:teuthology.orchestra.run.smithi150:mds.d> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T15:57:16.405 DEBUG:teuthology.orchestra.run.smithi150:mds.g> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T15:57:16.448 DEBUG:teuthology.orchestra.run.smithi150:mds.j> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T15:57:16.491 DEBUG:teuthology.orchestra.run.smithi156:mds.b> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T15:57:16.493 DEBUG:teuthology.orchestra.run.smithi156:mds.e> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T15:57:16.537 DEBUG:teuthology.orchestra.run.smithi156:mds.h> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T15:57:16.579 DEBUG:teuthology.orchestra.run.smithi156:mds.k> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T15:57:16.623 DEBUG:teuthology.orchestra.run.smithi191:mds.c> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T15:57:16.625 DEBUG:teuthology.orchestra.run.smithi191:mds.f> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T15:57:16.648 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:16 smithi191 ceph-mon[36784]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:16.648 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:16.648 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:16 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:16.649 DEBUG:teuthology.orchestra.run.smithi191:mds.i> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T15:57:16.692 DEBUG:teuthology.orchestra.run.smithi191:mds.l> sudo journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T15:57:16.735 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T15:57:16.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:57:16.760 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:16 smithi150 ceph-mon[31946]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:16.760 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:16.760 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:16 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:16 smithi156 ceph-mon[35919]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:16 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 -- 172.21.15.150:0/234461150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 msgr2=0x7f9048108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 --2- 172.21.15.150:0/234461150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f9048108b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9040009ca0 tx=0x7f904002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 -- 172.21.15.150:0/234461150 shutdown_connections 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 --2- 172.21.15.150:0/234461150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f9048108b40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 --2- 172.21.15.150:0/234461150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f9048101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 --2- 172.21.15.150:0/234461150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 0x7f9048100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 -- 172.21.15.150:0/234461150 >> 172.21.15.150:0/234461150 conn(0x7f90480fbc70 msgr2=0x7f90480fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 -- 172.21.15.150:0/234461150 shutdown_connections 2024-03-18T15:57:16.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 -- 172.21.15.150:0/234461150 wait complete. 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.842+0000 7f90507ba640 1 Processor -- start 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 -- start start 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 0x7f90480694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f904819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f904810c380 con 0x7f90481022b0 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f904810c200 con 0x7f9048100700 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f904810c500 con 0x7f90481018f0 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59126/0 (socket says 172.21.15.150:59126) 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 -- 172.21.15.150:0/4268166948 learned_addr learned my addr 172.21.15.150:0/4268166948 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904dd55640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f904819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904e556640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 0x7f90480694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:16.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 -- 172.21.15.150:0/4268166948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 msgr2=0x7f904819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f904819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 -- 172.21.15.150:0/4268166948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 msgr2=0x7f90480694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 0x7f90480694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 -- 172.21.15.150:0/4268166948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904819f6f0 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904dd55640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f904819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f904ed57640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9040037040 tx=0x7f9040032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040041020 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9040034d90 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040039040 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.843+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904819f980 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.844+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904819fe20 con 0x7f90481022b0 2024-03-18T15:57:16.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.845+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048107a20 con 0x7f90481022b0 2024-03-18T15:57:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.846+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9040044020 con 0x7f90481022b0 2024-03-18T15:57:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.846+0000 7f903f7fe640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 0x7f90340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.846+0000 7f904e556640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 0x7f90340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.846+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f90400f0820 con 0x7f90481022b0 2024-03-18T15:57:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.847+0000 7f904e556640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 0x7f90340a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9048101750 tx=0x7f904400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:16.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:16.850+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f904004b020 con 0x7f90481022b0 2024-03-18T15:57:17.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.027+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f904802d380 con 0x7f90481022b0 2024-03-18T15:57:17.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.028+0000 7f903f7fe640 1 -- 172.21.15.150:0/4268166948 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18512 (secure 0 0 0) 0x7f90400bde80 con 0x7f90481022b0 2024-03-18T15:57:17.029 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:17.029 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":82,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:07.237328+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 msgr2=0x7f90340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 0x7f90340a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9048101750 tx=0x7f904400b390 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 msgr2=0x7f904819ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9040037040 tx=0x7f9040032f20 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 shutdown_connections 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90340a2880 0x7f90340a4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90481022b0 0x7f904819ee30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90481018f0 0x7f904819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 --2- 172.21.15.150:0/4268166948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9048100700 0x7f90480694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 >> 172.21.15.150:0/4268166948 conn(0x7f90480fbc70 msgr2=0x7f9048104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 shutdown_connections 2024-03-18T15:57:17.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.030+0000 7f90507ba640 1 -- 172.21.15.150:0/4268166948 wait complete. 2024-03-18T15:57:17.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:57:17.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 -- 172.21.15.150:0/1938914879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541018f0 msgr2=0x7fa154101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 --2- 172.21.15.150:0/1938914879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541018f0 0x7fa154101d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa144009900 tx=0x7fa14402bc40 comp rx=0 tx=0).stop 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 -- 172.21.15.150:0/1938914879 shutdown_connections 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 --2- 172.21.15.150:0/1938914879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1541022b0 0x7fa154108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 --2- 172.21.15.150:0/1938914879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541018f0 0x7fa154101d70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 --2- 172.21.15.150:0/1938914879 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa154100700 0x7fa154100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 -- 172.21.15.150:0/1938914879 >> 172.21.15.150:0/1938914879 conn(0x7fa1540fbc90 msgr2=0x7fa1540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 -- 172.21.15.150:0/1938914879 shutdown_connections 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 -- 172.21.15.150:0/1938914879 wait complete. 2024-03-18T15:57:17.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.187+0000 7fa15b752640 1 Processor -- start 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- start start 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 0x7fa15406a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 0x7fa15419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa15410c360 con 0x7fa1541022b0 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa15410c1e0 con 0x7fa154100700 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa15410c4e0 con 0x7fa1541018f0 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59138/0 (socket says 172.21.15.150:59138) 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 -- 172.21.15.150:0/2838773528 learned_addr learned my addr 172.21.15.150:0/2838773528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa1594ee640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 0x7fa15406a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa158ced640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 0x7fa15419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 -- 172.21.15.150:0/2838773528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 msgr2=0x7fa15419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 0x7fa15419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 -- 172.21.15.150:0/2838773528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 msgr2=0x7fa15406a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 0x7fa15406a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 -- 172.21.15.150:0/2838773528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15419f6e0 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa1594ee640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 0x7fa15406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa158ced640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 0x7fa15419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa159cef640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00cce0 tx=0x7fa14c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c00f450 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa14c00fad0 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa15419f9d0 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c018c10 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.188+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa15419fea0 con 0x7fa1541022b0 2024-03-18T15:57:17.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.189+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa154068a20 con 0x7fa1541022b0 2024-03-18T15:57:17.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.191+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa14c00f5f0 con 0x7fa1541022b0 2024-03-18T15:57:17.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.191+0000 7fa14a7fc640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 0x7fa1400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.191+0000 7fa1594ee640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 0x7fa1400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.191+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa14c0cedb0 con 0x7fa1541022b0 2024-03-18T15:57:17.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.192+0000 7fa1594ee640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 0x7fa1400a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa154101750 tx=0x7fa150005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:17.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.195+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa14c014030 con 0x7fa1541022b0 2024-03-18T15:57:17.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.370+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa15402d1e0 con 0x7fa1541022b0 2024-03-18T15:57:17.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.370+0000 7fa14a7fc640 1 -- 172.21.15.150:0/2838773528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fa14c09c380 con 0x7fa1541022b0 2024-03-18T15:57:17.372 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 msgr2=0x7fa1400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 0x7fa1400a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa154101750 tx=0x7fa150005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 msgr2=0x7fa15419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00cce0 tx=0x7fa14c00a620 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 shutdown_connections 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1400a2800 0x7fa1400a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1541022b0 0x7fa15419ee20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1541018f0 0x7fa15419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 --2- 172.21.15.150:0/2838773528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa154100700 0x7fa15406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 >> 172.21.15.150:0/2838773528 conn(0x7fa1540fbc90 msgr2=0x7fa154104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:17.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 shutdown_connections 2024-03-18T15:57:17.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.372+0000 7fa15b752640 1 -- 172.21.15.150:0/2838773528 wait complete. 2024-03-18T15:57:17.385 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.485+0000 7f666f0ff640 1 -- 172.21.15.150:0/2484339267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681022b0 msgr2=0x7f6668108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.485+0000 7f666f0ff640 1 --2- 172.21.15.150:0/2484339267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681022b0 0x7f6668108b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6660009ca0 tx=0x7f666002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- 172.21.15.150:0/2484339267 shutdown_connections 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- 172.21.15.150:0/2484339267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681022b0 0x7f6668108b40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- 172.21.15.150:0/2484339267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681018f0 0x7f6668101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- 172.21.15.150:0/2484339267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 0x7f6668100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- 172.21.15.150:0/2484339267 >> 172.21.15.150:0/2484339267 conn(0x7f66680fbc70 msgr2=0x7f66680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- 172.21.15.150:0/2484339267 shutdown_connections 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- 172.21.15.150:0/2484339267 wait complete. 2024-03-18T15:57:17.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 Processor -- start 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- start start 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 0x7f6668197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 0x7f666819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f666810d4a0 con 0x7f66681018f0 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f666810d320 con 0x7f6668100700 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666f0ff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f666810d620 con 0x7f66681022b0 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f6667fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f6667fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59164/0 (socket says 172.21.15.150:59164) 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f6667fff640 1 -- 172.21.15.150:0/1609916419 learned_addr learned my addr 172.21.15.150:0/1609916419 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.486+0000 7f666ce9b640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 0x7f6668197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f666d69c640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 0x7f666819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 -- 172.21.15.150:0/1609916419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 msgr2=0x7f666819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 0x7f666819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 -- 172.21.15.150:0/1609916419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 msgr2=0x7f6668197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 0x7f6668197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 -- 172.21.15.150:0/1609916419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666819f750 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f666d69c640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 0x7f666819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6667fff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f665400d770 tx=0x7f665400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654017070 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6654005500 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666819fa40 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654004e00 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.487+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f666819ff40 con 0x7f66681018f0 2024-03-18T15:57:17.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.488+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66681079c0 con 0x7f66681018f0 2024-03-18T15:57:17.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.489+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6654004fa0 con 0x7f66681018f0 2024-03-18T15:57:17.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.490+0000 7f6665ffb640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:17.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.490+0000 7f666ce9b640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:17.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.490+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f66540cdff0 con 0x7f66681018f0 2024-03-18T15:57:17.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.490+0000 7f666ce9b640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 0x7f66580a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6668101750 tx=0x7f665c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:17.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.494+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f665409b530 con 0x7f66681018f0 2024-03-18T15:57:17.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:57:17.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: Saving service mds.all spec with placement smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-18T15:57:17.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:17.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: Deploying daemon mds.l on smithi191 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4268166948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:17.703 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:17 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2838773528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:17.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.711+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f666802d1b0 con 0x7f66681018f0 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: Saving service mds.all spec with placement smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: Deploying daemon mds.l on smithi191 2024-03-18T15:57:17.714 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4268166948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:17.714 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:17 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2838773528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:17.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.712+0000 7f6665ffb640 1 -- 172.21.15.150:0/1609916419 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f665409ae30 con 0x7f66681018f0 2024-03-18T15:57:17.714 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:17.714 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T15:57:17.715 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 1 2024-03-18T15:57:17.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 msgr2=0x7f66580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 0x7f66580a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6668101750 tx=0x7f665c00b390 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 msgr2=0x7f6668198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f665400d770 tx=0x7f665400dc30 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 shutdown_connections 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66580a2800 0x7f66580a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66681022b0 0x7f666819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66681018f0 0x7f6668198070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 --2- 172.21.15.150:0/1609916419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6668100700 0x7f6668197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 >> 172.21.15.150:0/1609916419 conn(0x7f66680fbc70 msgr2=0x7f6668104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 shutdown_connections 2024-03-18T15:57:17.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:17.714+0000 7f666f0ff640 1 -- 172.21.15.150:0/1609916419 wait complete. 2024-03-18T15:57:17.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:57:17.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: Saving service mds.all spec with placement smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-18T15:57:17.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: Deploying daemon mds.l on smithi191 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4268166948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:17.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:17 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2838773528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:18.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:18 smithi150 ceph-mon[31946]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:18.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:18 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1609916419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:18.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:18 smithi191 ceph-mon[36784]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:18.897 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:18 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1609916419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:18.898 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:18 smithi191 systemd[1]: Starting Ceph mds.l for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:18 smithi156 ceph-mon[35919]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:18 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1609916419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:19.159 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:18 smithi191 podman[66984]: 2024-03-18 15:57:18.799068267 +0000 UTC m=+0.017346501 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:19.159 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:18 smithi191 podman[66984]: 2024-03-18 15:57:18.921205 +0000 UTC m=+0.139483234 container create b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T15:57:19.159 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 podman[66984]: 2024-03-18 15:57:19.158561666 +0000 UTC m=+0.376839890 container init b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T15:57:19.462 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 podman[66984]: 2024-03-18 15:57:19.161348782 +0000 UTC m=+0.379627023 container start b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:57:19.462 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 ceph-mds[67003]: -- 172.21.15.191:0/960071718 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56112c62fa40 con 0x56112c602800 2024-03-18T15:57:19.462 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: starting mds.l at 2024-03-18T15:57:19.462 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 bash[66984]: b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da 2024-03-18T15:57:19.463 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 15:57:19 smithi191 systemd[1]: Started Ceph mds.l for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:19.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:19.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 -- 172.21.15.150:0/4234357638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664101970 msgr2=0x7f3664101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:19.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 --2- 172.21.15.150:0/4234357638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664101970 0x7f3664101df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3654009900 tx=0x7f365402bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:19.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 -- 172.21.15.150:0/4234357638 shutdown_connections 2024-03-18T15:57:19.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 --2- 172.21.15.150:0/4234357638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664102330 0x7f3664108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:19.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 --2- 172.21.15.150:0/4234357638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664101970 0x7f3664101df0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 --2- 172.21.15.150:0/4234357638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 0x7f3664100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 -- 172.21.15.150:0/4234357638 >> 172.21.15.150:0/4234357638 conn(0x7f36640fbd10 msgr2=0x7f36640fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 -- 172.21.15.150:0/4234357638 shutdown_connections 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 -- 172.21.15.150:0/4234357638 wait complete. 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.840+0000 7f3669737640 1 Processor -- start 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 -- start start 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 0x7f3664197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664101970 0x7f3664198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f366410d4b0 con 0x7f3664102330 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f366410d330 con 0x7f3664100780 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3669737640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f366410d630 con 0x7f3664101970 2024-03-18T15:57:19.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59184/0 (socket says 172.21.15.150:59184) 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 -- 172.21.15.150:0/703448188 learned_addr learned my addr 172.21.15.150:0/703448188 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f3662ffd640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 0x7f3664197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36627fc640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664101970 0x7f3664198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 -- 172.21.15.150:0/703448188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664101970 msgr2=0x7f3664198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664101970 0x7f3664198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 -- 172.21.15.150:0/703448188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 msgr2=0x7f3664197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 0x7f3664197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 -- 172.21.15.150:0/703448188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366419f700 con 0x7f3664102330 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.841+0000 7f36637fe640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f365c00eaa0 tx=0x7f365c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.842+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c00cdb0 con 0x7f3664102330 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.842+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366419f9f0 con 0x7f3664102330 2024-03-18T15:57:19.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.842+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f365c005020 con 0x7f3664102330 2024-03-18T15:57:19.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.842+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365c0116d0 con 0x7f3664102330 2024-03-18T15:57:19.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.842+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366419ff90 con 0x7f3664102330 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.843+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3664107b40 con 0x7f3664102330 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.844+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f365c0054d0 con 0x7f3664102330 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.844+0000 7f364bfff640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 0x7f364c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.845+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f365c0ce780 con 0x7f3664102330 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.845+0000 7f3662ffd640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 0x7f364c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:19.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.845+0000 7f3662ffd640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 0x7f364c0a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f36641017d0 tx=0x7f3658005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:19.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:19.849+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f365c016030 con 0x7f3664102330 2024-03-18T15:57:20.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.060+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f366402d1b0 con 0x7f3664102330 2024-03-18T15:57:20.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.061+0000 7f364bfff640 1 -- 172.21.15.150:0/703448188 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f365c09bcc0 con 0x7f3664102330 2024-03-18T15:57:20.062 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:20.062 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:20.062 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 2 2024-03-18T15:57:20.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 msgr2=0x7f364c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:20.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 0x7f364c0a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f36641017d0 tx=0x7f3658005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:20.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 msgr2=0x7f366419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:20.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f365c00eaa0 tx=0x7f365c00ef60 comp rx=0 tx=0).stop 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 shutdown_connections 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f364c0a2880 0x7f364c0a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3664102330 0x7f366419f090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3664101970 0x7f3664198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 --2- 172.21.15.150:0/703448188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3664100780 0x7f3664197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 >> 172.21.15.150:0/703448188 conn(0x7f36640fbd10 msgr2=0x7f3664104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 shutdown_connections 2024-03-18T15:57:20.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:20.063+0000 7f3669737640 1 -- 172.21.15.150:0/703448188 wait complete. 2024-03-18T15:57:20.229 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6832/2692248421,v1:172.21.15.191:6833/2692248421] up:boot 2024-03-18T15:57:20.229 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: fsmap 1 up:standby 2024-03-18T15:57:20.229 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:57:20.229 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.229 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: Deploying daemon mds.k on smithi156 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:20.230 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:20 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/703448188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6832/2692248421,v1:172.21.15.191:6833/2692248421] up:boot 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: fsmap 1 up:standby 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:20.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:20.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:20.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: Deploying daemon mds.k on smithi156 2024-03-18T15:57:20.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:20.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:20 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/703448188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:20.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6832/2692248421,v1:172.21.15.191:6833/2692248421] up:boot 2024-03-18T15:57:20.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: fsmap 1 up:standby 2024-03-18T15:57:20.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T15:57:20.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: Deploying daemon mds.k on smithi156 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:20.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:20 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/703448188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:21.755 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:21 smithi156 systemd[1]: Starting Ceph mds.k for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:22.018 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:21 smithi156 podman[65395]: 2024-03-18 15:57:21.657884154 +0000 UTC m=+0.018199328 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:22.018 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:21 smithi156 podman[65395]: 2024-03-18 15:57:21.778171573 +0000 UTC m=+0.138486776 container create 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T15:57:22.018 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:21 smithi156 podman[65395]: 2024-03-18 15:57:21.997333146 +0000 UTC m=+0.357648307 container init 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T15:57:22.018 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:21 smithi156 podman[65395]: 2024-03-18 15:57:21.999917659 +0000 UTC m=+0.360232821 container start 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T15:57:22.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:22.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.186+0000 7f253aca1640 1 -- 172.21.15.150:0/2178803068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341022b0 msgr2=0x7f2534108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:22.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.186+0000 7f253aca1640 1 --2- 172.21.15.150:0/2178803068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341022b0 0x7f2534108b40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f252c009900 tx=0x7f252c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:22.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- 172.21.15.150:0/2178803068 shutdown_connections 2024-03-18T15:57:22.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- 172.21.15.150:0/2178803068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341022b0 0x7f2534108b40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- 172.21.15.150:0/2178803068 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25341018f0 0x7f2534101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- 172.21.15.150:0/2178803068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2534100700 0x7f2534100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- 172.21.15.150:0/2178803068 >> 172.21.15.150:0/2178803068 conn(0x7f25340fbc70 msgr2=0x7f25340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- 172.21.15.150:0/2178803068 shutdown_connections 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- 172.21.15.150:0/2178803068 wait complete. 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 Processor -- start 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- start start 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2534100700 0x7f2534197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25341022b0 0x7f253419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f253410c360 con 0x7f25341018f0 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f253410c1e0 con 0x7f25341022b0 2024-03-18T15:57:22.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f253aca1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f253410c4e0 con 0x7f2534100700 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f2533fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f2533fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59212/0 (socket says 172.21.15.150:59212) 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.187+0000 7f2533fff640 1 -- 172.21.15.150:0/4294690427 learned_addr learned my addr 172.21.15.150:0/4294690427 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2538a3d640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2534100700 0x7f2534197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f253923e640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25341022b0 0x7f253419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 -- 172.21.15.150:0/4294690427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2534100700 msgr2=0x7f2534197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2534100700 0x7f2534197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 -- 172.21.15.150:0/4294690427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25341022b0 msgr2=0x7f253419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25341022b0 0x7f253419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 -- 172.21.15.150:0/4294690427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253419f6e0 con 0x7f25341018f0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2533fff640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f252400cce0 tx=0x7f252400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524004b20 con 0x7f25341018f0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2524004cc0 con 0x7f25341018f0 2024-03-18T15:57:22.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253419f9d0 con 0x7f25341018f0 2024-03-18T15:57:22.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252400f7d0 con 0x7f25341018f0 2024-03-18T15:57:22.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.188+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253419ff70 con 0x7f25341018f0 2024-03-18T15:57:22.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.189+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25341a0200 con 0x7f25341018f0 2024-03-18T15:57:22.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.190+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f252400f970 con 0x7f25341018f0 2024-03-18T15:57:22.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.191+0000 7f2531ffb640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 0x7f25200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:22.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.191+0000 7f2538a3d640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 0x7f25200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:22.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.191+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f25240ce600 con 0x7f25341018f0 2024-03-18T15:57:22.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.191+0000 7f2538a3d640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 0x7f25200a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2534101750 tx=0x7f251c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:22.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.196+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2524014030 con 0x7f25341018f0 2024-03-18T15:57:22.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.427+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25341a03d0 con 0x7f25341018f0 2024-03-18T15:57:22.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.427+0000 7f2531ffb640 1 -- 172.21.15.150:0/4294690427 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f252409bb40 con 0x7f25341018f0 2024-03-18T15:57:22.429 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:22.429 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:22.429 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 2 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.429+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 msgr2=0x7f25200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.429+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 0x7f25200a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2534101750 tx=0x7f251c005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.429+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 msgr2=0x7f2534198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.429+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f252400cce0 tx=0x7f252400a620 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 shutdown_connections 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f25200a2880 0x7f25200a4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25341022b0 0x7f253419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25341018f0 0x7f2534198000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 --2- 172.21.15.150:0/4294690427 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2534100700 0x7f2534197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 >> 172.21.15.150:0/4294690427 conn(0x7f25340fbc70 msgr2=0x7f25340fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:22.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 shutdown_connections 2024-03-18T15:57:22.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:22.430+0000 7f253aca1640 1 -- 172.21.15.150:0/4294690427 wait complete. 2024-03-18T15:57:22.492 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mds[65413]: -- 172.21.15.156:0/3429324676 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af8536da40 con 0x55af85340c00 2024-03-18T15:57:22.492 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k[65409]: starting mds.k at 2024-03-18T15:57:22.492 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:22 smithi156 bash[65395]: 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c 2024-03-18T15:57:22.493 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 15:57:22 smithi156 systemd[1]: Started Ceph mds.k for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:22.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:22.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:22 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4294690427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:22 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4294690427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:22 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4294690427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:23.535 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:23 smithi156 ceph-mon[35919]: Deploying daemon mds.j on smithi150 2024-03-18T15:57:23.535 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:23 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6832/925824611,v1:172.21.15.156:6833/925824611] up:boot 2024-03-18T15:57:23.535 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:23 smithi156 ceph-mon[35919]: fsmap 2 up:standby 2024-03-18T15:57:23.535 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:23 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:57:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:23 smithi150 ceph-mon[31946]: Deploying daemon mds.j on smithi150 2024-03-18T15:57:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:23 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6832/925824611,v1:172.21.15.156:6833/925824611] up:boot 2024-03-18T15:57:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:23 smithi150 ceph-mon[31946]: fsmap 2 up:standby 2024-03-18T15:57:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:23 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:57:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:23 smithi191 ceph-mon[36784]: Deploying daemon mds.j on smithi150 2024-03-18T15:57:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:23 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6832/925824611,v1:172.21.15.156:6833/925824611] up:boot 2024-03-18T15:57:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:23 smithi191 ceph-mon[36784]: fsmap 2 up:standby 2024-03-18T15:57:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:23 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T15:57:24.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 -- 172.21.15.150:0/1786660314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac100240 msgr2=0x7f68ac109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/1786660314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac100240 0x7f68ac109df0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009be0 tx=0x7f68a402bef0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 -- 172.21.15.150:0/1786660314 shutdown_connections 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/1786660314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac100240 0x7f68ac109df0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/1786660314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac0ff880 0x7f68ac0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/1786660314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac102e10 0x7f68ac103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 -- 172.21.15.150:0/1786660314 >> 172.21.15.150:0/1786660314 conn(0x7f68ac0fb420 msgr2=0x7f68ac0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 -- 172.21.15.150:0/1786660314 shutdown_connections 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 -- 172.21.15.150:0/1786660314 wait complete. 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.559+0000 7f68b2e0b640 1 Processor -- start 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 -- start start 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac100240 0x7f68ac197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 0x7f68ac19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10d4c0 con 0x7f68ac0ff880 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10d340 con 0x7f68ac102e10 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b2e0b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10d640 con 0x7f68ac100240 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47864/0 (socket says 172.21.15.150:47864) 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 -- 172.21.15.150:0/3510222498 learned_addr learned my addr 172.21.15.150:0/3510222498 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68abfff640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac100240 0x7f68ac197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b13a8640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 0x7f68ac19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 -- 172.21.15.150:0/3510222498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac100240 msgr2=0x7f68ac197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac100240 0x7f68ac197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 -- 172.21.15.150:0/3510222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 msgr2=0x7f68ac19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 0x7f68ac19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 -- 172.21.15.150:0/3510222498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac19f6d0 con 0x7f68ac0ff880 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b13a8640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 0x7f68ac19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.560+0000 7f68b0ba7640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e9a0 tx=0x7f68a000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.561+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a000cd80 con 0x7f68ac0ff880 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.561+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68a0010910 con 0x7f68ac0ff880 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.561+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0018a30 con 0x7f68ac0ff880 2024-03-18T15:57:24.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.561+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac19f9c0 con 0x7f68ac0ff880 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.561+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac19ff10 con 0x7f68ac0ff880 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.562+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ac108c70 con 0x7f68ac0ff880 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.564+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f68a0010ac0 con 0x7f68ac0ff880 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.564+0000 7f68a9ffb640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 0x7f689c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.564+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f68a00cf010 con 0x7f68ac0ff880 2024-03-18T15:57:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.564+0000 7f68abfff640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 0x7f689c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.565+0000 7f68abfff640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 0x7f689c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68ac198fc0 tx=0x7f689800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:24.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.568+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f68a0026070 con 0x7f68ac0ff880 2024-03-18T15:57:24.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:24 smithi150 ceph-mon[31946]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:24.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:24 smithi156 ceph-mon[35919]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:24.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.792+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68ac02d370 con 0x7f68ac0ff880 2024-03-18T15:57:24.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.793+0000 7f68a9ffb640 1 -- 172.21.15.150:0/3510222498 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f68a009c550 con 0x7f68ac0ff880 2024-03-18T15:57:24.794 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:24.794 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:24.794 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 3 2024-03-18T15:57:24.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 msgr2=0x7f689c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:24.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 0x7f689c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68ac198fc0 tx=0x7f689800b390 comp rx=0 tx=0).stop 2024-03-18T15:57:24.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 msgr2=0x7f68ac197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e9a0 tx=0x7f68a000ee60 comp rx=0 tx=0).stop 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 shutdown_connections 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f689c0a2800 0x7f689c0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68ac102e10 0x7f68ac19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68ac100240 0x7f68ac197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 --2- 172.21.15.150:0/3510222498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68ac0ff880 0x7f68ac197ab0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 >> 172.21.15.150:0/3510222498 conn(0x7f68ac0fb420 msgr2=0x7f68ac105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 shutdown_connections 2024-03-18T15:57:24.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:24.795+0000 7f68b2e0b640 1 -- 172.21.15.150:0/3510222498 wait complete. 2024-03-18T15:57:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:24 smithi191 ceph-mon[36784]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:25.342 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 15:57:25 smithi150 systemd[1]: Started Ceph mds.j for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3510222498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:25.769 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:25 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3510222498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:25 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3510222498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:25 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:26.803 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:26 smithi150 ceph-mon[31946]: Deploying daemon mds.d on smithi150 2024-03-18T15:57:26.803 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:26 smithi150 ceph-mon[31946]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:26.803 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:26 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.150:6834/3457359826,v1:172.21.15.150:6835/3457359826] up:boot 2024-03-18T15:57:26.803 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:26 smithi150 ceph-mon[31946]: fsmap 3 up:standby 2024-03-18T15:57:26.803 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:26 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:57:26.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/4266180480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc14104bf0 msgr2=0x7fcc14106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/4266180480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc14104bf0 0x7fcc14106fe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009900 tx=0x7fcc0402bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/4266180480 shutdown_connections 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/4266180480 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc14109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/4266180480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc14104bf0 0x7fcc14106fe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/4266180480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc141022c0 0x7fcc141046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.916+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/4266180480 >> 172.21.15.150:0/4266180480 conn(0x7fcc140fbc70 msgr2=0x7fcc140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/4266180480 shutdown_connections 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/4266180480 wait complete. 2024-03-18T15:57:26.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 Processor -- start 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- start start 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14104bf0 0x7fcc14100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc141007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410d460 con 0x7fcc141022c0 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410d2e0 con 0x7fcc14104bf0 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc1bd5b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410d5e0 con 0x7fcc14107520 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc19af7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.917+0000 7fcc19af7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47902/0 (socket says 172.21.15.150:47902) 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 -- 172.21.15.150:0/92076273 learned_addr learned my addr 172.21.15.150:0/92076273 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc192f6640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14104bf0 0x7fcc14100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc1a2f8640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc141007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 -- 172.21.15.150:0/92076273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 msgr2=0x7fcc141007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:26.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc141007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 -- 172.21.15.150:0/92076273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14104bf0 msgr2=0x7fcc14100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14104bf0 0x7fcc14100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 -- 172.21.15.150:0/92076273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc14100fb0 con 0x7fcc141022c0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc1a2f8640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc141007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc19af7640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c890 tx=0x7fcc1000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004220 con 0x7fcc141022c0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc100043c0 con 0x7fcc141022c0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc141a1c10 con 0x7fcc141022c0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100049e0 con 0x7fcc141022c0 2024-03-18T15:57:26.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.918+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc141a2180 con 0x7fcc141022c0 2024-03-18T15:57:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.919+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc140fd760 con 0x7fcc141022c0 2024-03-18T15:57:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.920+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fcc100079b0 con 0x7fcc141022c0 2024-03-18T15:57:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.921+0000 7fcc0affd640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 0x7fcc000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.921+0000 7fcc192f6640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 0x7fcc000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.921+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcc100ce5b0 con 0x7fcc141022c0 2024-03-18T15:57:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.921+0000 7fcc192f6640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 0x7fcc000a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc04004f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:26.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:26.925+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc10014030 con 0x7fcc141022c0 2024-03-18T15:57:26.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:26 smithi191 ceph-mon[36784]: Deploying daemon mds.d on smithi150 2024-03-18T15:57:26.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:26 smithi191 ceph-mon[36784]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:26.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:26 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.150:6834/3457359826,v1:172.21.15.150:6835/3457359826] up:boot 2024-03-18T15:57:26.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:26 smithi191 ceph-mon[36784]: fsmap 3 up:standby 2024-03-18T15:57:26.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:26 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:57:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:26 smithi156 ceph-mon[35919]: Deploying daemon mds.d on smithi150 2024-03-18T15:57:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:26 smithi156 ceph-mon[35919]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:26 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.150:6834/3457359826,v1:172.21.15.150:6835/3457359826] up:boot 2024-03-18T15:57:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:26 smithi156 ceph-mon[35919]: fsmap 3 up:standby 2024-03-18T15:57:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:26 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T15:57:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.161+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc1402d420 con 0x7fcc141022c0 2024-03-18T15:57:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.161+0000 7fcc0affd640 1 -- 172.21.15.150:0/92076273 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fcc1009baf0 con 0x7fcc141022c0 2024-03-18T15:57:27.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:27.163 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 4 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.163+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 msgr2=0x7fcc000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.163+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 0x7fcc000a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc04004f10 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.163+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 msgr2=0x7fcc14101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.163+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c890 tx=0x7fcc1000cd50 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 shutdown_connections 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcc000a2880 0x7fcc000a4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc14107520 0x7fcc141007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc14104bf0 0x7fcc14100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 --2- 172.21.15.150:0/92076273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc141022c0 0x7fcc14101b90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 >> 172.21.15.150:0/92076273 conn(0x7fcc140fbc70 msgr2=0x7fcc14102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:27.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 shutdown_connections 2024-03-18T15:57:27.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:27.164+0000 7fcc1bd5b640 1 -- 172.21.15.150:0/92076273 wait complete. 2024-03-18T15:57:27.342 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 systemd[1]: Starting Ceph mds.d for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:27.750 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:27 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/92076273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:27.751 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 podman[70463]: 2024-03-18 15:57:27.385848738 +0000 UTC m=+0.017647391 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:27.751 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 podman[70463]: 2024-03-18 15:57:27.521692067 +0000 UTC m=+0.153490714 container create 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T15:57:27.751 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 podman[70463]: 2024-03-18 15:57:27.750184132 +0000 UTC m=+0.381982788 container init 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T15:57:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:27 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/92076273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:27 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/92076273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:28.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 podman[70463]: 2024-03-18 15:57:27.752928687 +0000 UTC m=+0.384727336 container start 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:28.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 ceph-mds[70482]: -- 172.21.15.150:0/1176460350 <== mon.0 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2d1b3fa40 con 0x55d2d1b13000 2024-03-18T15:57:28.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d[70478]: starting mds.d at 2024-03-18T15:57:28.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 bash[70463]: 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 2024-03-18T15:57:28.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 15:57:27 smithi150 systemd[1]: Started Ceph mds.d for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:28.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:28.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:28.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:28.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:28 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:28 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:28.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:28 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:29.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 -- 172.21.15.150:0/2604515752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac1022d0 msgr2=0x7f28ac108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 --2- 172.21.15.150:0/2604515752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac1022d0 0x7f28ac108b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009ca0 tx=0x7f28a402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 -- 172.21.15.150:0/2604515752 shutdown_connections 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 --2- 172.21.15.150:0/2604515752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac1022d0 0x7f28ac108b60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 --2- 172.21.15.150:0/2604515752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac101910 0x7f28ac101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 --2- 172.21.15.150:0/2604515752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac100720 0x7f28ac100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 -- 172.21.15.150:0/2604515752 >> 172.21.15.150:0/2604515752 conn(0x7f28ac0fbc90 msgr2=0x7f28ac0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 -- 172.21.15.150:0/2604515752 shutdown_connections 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 -- 172.21.15.150:0/2604515752 wait complete. 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.295+0000 7f28b3427640 1 Processor -- start 2024-03-18T15:57:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- start start 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 0x7f28ac19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac1022d0 0x7f28ac19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10c380 con 0x7f28ac100720 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10c200 con 0x7f28ac101910 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10c500 con 0x7f28ac1022d0 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47932/0 (socket says 172.21.15.150:47932) 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 -- 172.21.15.150:0/1996253485 learned_addr learned my addr 172.21.15.150:0/1996253485 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b19c4640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac1022d0 0x7f28ac19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b09c2640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 0x7f28ac19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 -- 172.21.15.150:0/1996253485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac1022d0 msgr2=0x7f28ac19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac1022d0 0x7f28ac19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 -- 172.21.15.150:0/1996253485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 msgr2=0x7f28ac19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 0x7f28ac19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 -- 172.21.15.150:0/1996253485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac19f710 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b09c2640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 0x7f28ac19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b11c3640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9c0 tx=0x7f28a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a800cd80 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a80035f0 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac19f9a0 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8010690 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.296+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac19fef0 con 0x7f28ac100720 2024-03-18T15:57:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.298+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac068a20 con 0x7f28ac100720 2024-03-18T15:57:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.299+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f28a80037a0 con 0x7f28ac100720 2024-03-18T15:57:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.299+0000 7f28a27fc640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 0x7f28980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.299+0000 7f28b09c2640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 0x7f28980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.299+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f28a80ceae0 con 0x7f28ac100720 2024-03-18T15:57:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.300+0000 7f28b09c2640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 0x7f28980a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f28ac06a6f0 tx=0x7f289c005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.304+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f28a809c020 con 0x7f28ac100720 2024-03-18T15:57:29.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.515+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28ac107a90 con 0x7f28ac100720 2024-03-18T15:57:29.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.516+0000 7f28a27fc640 1 -- 172.21.15.150:0/1996253485 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3877 (secure 0 0 0) 0x7f28ac107a90 con 0x7f28ac100720 2024-03-18T15:57:29.517 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:29.517 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:29.517 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 5 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 msgr2=0x7f28980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 0x7f28980a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f28ac06a6f0 tx=0x7f289c005e50 comp rx=0 tx=0).stop 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 msgr2=0x7f28ac06a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9c0 tx=0x7f28a800ee80 comp rx=0 tx=0).stop 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 shutdown_connections 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28980a2800 0x7f28980a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28ac1022d0 0x7f28ac19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28ac101910 0x7f28ac19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 --2- 172.21.15.150:0/1996253485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac100720 0x7f28ac06a980 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:29.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 >> 172.21.15.150:0/1996253485 conn(0x7f28ac0fbc90 msgr2=0x7f28ac104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:29.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 shutdown_connections 2024-03-18T15:57:29.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:29.518+0000 7f28b3427640 1 -- 172.21.15.150:0/1996253485 wait complete. 2024-03-18T15:57:29.813 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:29 smithi156 ceph-mon[35919]: Deploying daemon mds.e on smithi156 2024-03-18T15:57:29.814 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:29 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.150:6836/3172588678,v1:172.21.15.150:6837/3172588678] up:boot 2024-03-18T15:57:29.814 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:29 smithi156 ceph-mon[35919]: fsmap 4 up:standby 2024-03-18T15:57:29.814 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:29 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:57:29.814 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:29 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1996253485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:29 smithi150 ceph-mon[31946]: Deploying daemon mds.e on smithi156 2024-03-18T15:57:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:29 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.150:6836/3172588678,v1:172.21.15.150:6837/3172588678] up:boot 2024-03-18T15:57:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:29 smithi150 ceph-mon[31946]: fsmap 4 up:standby 2024-03-18T15:57:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:29 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:57:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:29 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1996253485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:29 smithi191 ceph-mon[36784]: Deploying daemon mds.e on smithi156 2024-03-18T15:57:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:29 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.150:6836/3172588678,v1:172.21.15.150:6837/3172588678] up:boot 2024-03-18T15:57:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:29 smithi191 ceph-mon[36784]: fsmap 4 up:standby 2024-03-18T15:57:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:29 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T15:57:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:29 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1996253485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:30.518 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 podman[65721]: 2024-03-18 15:57:30.441412891 +0000 UTC m=+0.125099662 container create 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:57:30.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:30 smithi150 ceph-mon[31946]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:30.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:30 smithi191 ceph-mon[36784]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:31.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:30 smithi156 ceph-mon[35919]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:31.018 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 podman[65721]: 2024-03-18 15:57:30.661287324 +0000 UTC m=+0.344974098 container init 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:31.019 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 podman[65721]: 2024-03-18 15:57:30.664400034 +0000 UTC m=+0.348086816 container start 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:57:31.019 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 ceph-mds[65740]: -- 172.21.15.156:0/3373426800 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645340b9a40 con 0x56453408c800 2024-03-18T15:57:31.019 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: starting mds.e at 2024-03-18T15:57:31.019 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 bash[65721]: 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 2024-03-18T15:57:31.019 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 15:57:30 smithi156 systemd[1]: Started Ceph mds.e for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:31.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:31.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 -- 172.21.15.150:0/245546770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 msgr2=0x7f1210108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 --2- 172.21.15.150:0/245546770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f1210108b40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1208009ca0 tx=0x7f120802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 -- 172.21.15.150:0/245546770 shutdown_connections 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 --2- 172.21.15.150:0/245546770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f1210108b40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 --2- 172.21.15.150:0/245546770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f12101018f0 0x7f1210101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 --2- 172.21.15.150:0/245546770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1210100700 0x7f1210100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 -- 172.21.15.150:0/245546770 >> 172.21.15.150:0/245546770 conn(0x7f12100fbc70 msgr2=0x7f12100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 -- 172.21.15.150:0/245546770 shutdown_connections 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.635+0000 7f12170a1640 1 -- 172.21.15.150:0/245546770 wait complete. 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 Processor -- start 2024-03-18T15:57:31.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 -- start start 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1210100700 0x7f1210197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12101018f0 0x7f1210198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f121010c380 con 0x7f12101022b0 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f121010c200 con 0x7f12101018f0 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f12170a1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f121010c500 con 0x7f1210100700 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f121563e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f121563e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47956/0 (socket says 172.21.15.150:47956) 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f121563e640 1 -- 172.21.15.150:0/3182083534 learned_addr learned my addr 172.21.15.150:0/3182083534 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f120ffff640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12101018f0 0x7f1210198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.636+0000 7f1214e3d640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1210100700 0x7f1210197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 -- 172.21.15.150:0/3182083534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1210100700 msgr2=0x7f1210197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1210100700 0x7f1210197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 -- 172.21.15.150:0/3182083534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12101018f0 msgr2=0x7f1210198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12101018f0 0x7f1210198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 -- 172.21.15.150:0/3182083534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121019f730 con 0x7f12101022b0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f121563e640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1208037040 tx=0x7f1208032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208041020 con 0x7f12101022b0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1208034d90 con 0x7f12101022b0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121019f9c0 con 0x7f12101022b0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208039040 con 0x7f12101022b0 2024-03-18T15:57:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.637+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121019feb0 con 0x7f12101022b0 2024-03-18T15:57:31.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.638+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12101a01a0 con 0x7f12101022b0 2024-03-18T15:57:31.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.640+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1208044020 con 0x7f12101022b0 2024-03-18T15:57:31.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.640+0000 7f120dffb640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 0x7f12000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:31.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.640+0000 7f1214e3d640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 0x7f12000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:31.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.641+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f12080f04a0 con 0x7f12101022b0 2024-03-18T15:57:31.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.641+0000 7f1214e3d640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 0x7f12000a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1210101750 tx=0x7f120400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:31.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.645+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f120804b020 con 0x7f12101022b0 2024-03-18T15:57:31.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.866+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12101a0490 con 0x7f12101022b0 2024-03-18T15:57:31.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.867+0000 7f120dffb640 1 -- 172.21.15.150:0/3182083534 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4705 (secure 0 0 0) 0x7f12080bd9e0 con 0x7f12101022b0 2024-03-18T15:57:31.869 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:31.869 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T15:57:31.869 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 6 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 msgr2=0x7f12000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 0x7f12000a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1210101750 tx=0x7f120400b390 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 msgr2=0x7f121019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1208037040 tx=0x7f1208032f20 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 shutdown_connections 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f12000a2800 0x7f12000a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12101022b0 0x7f121019f0c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12101018f0 0x7f1210198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 --2- 172.21.15.150:0/3182083534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1210100700 0x7f1210197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.869+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 >> 172.21.15.150:0/3182083534 conn(0x7f12100fbc70 msgr2=0x7f12100fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.870+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 shutdown_connections 2024-03-18T15:57:31.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:31.870+0000 7f12170a1640 1 -- 172.21.15.150:0/3182083534 wait complete. 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: Deploying daemon mds.f on smithi191 2024-03-18T15:57:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:32.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6834/4268627742,v1:172.21.15.156:6835/4268627742] up:boot 2024-03-18T15:57:32.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: fsmap 5 up:standby 2024-03-18T15:57:32.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:31 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:57:32.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: Deploying daemon mds.f on smithi191 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6834/4268627742,v1:172.21.15.156:6835/4268627742] up:boot 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: fsmap 5 up:standby 2024-03-18T15:57:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:31 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:57:32.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: Deploying daemon mds.f on smithi191 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6834/4268627742,v1:172.21.15.156:6835/4268627742] up:boot 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: fsmap 5 up:standby 2024-03-18T15:57:32.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:31 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T15:57:32.925 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:32 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3182083534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:33.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:32 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3182083534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:33.213 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 systemd[1]: Starting Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:33.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:32 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3182083534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:33.680 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 podman[67293]: 2024-03-18 15:57:33.222848658 +0000 UTC m=+0.017607445 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:33.680 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 podman[67293]: 2024-03-18 15:57:33.383068993 +0000 UTC m=+0.177827785 container create 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T15:57:33.680 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 podman[67293]: 2024-03-18 15:57:33.679226206 +0000 UTC m=+0.473984992 container init 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T15:57:33.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:33 smithi191 ceph-mon[36784]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:33.962 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 podman[67293]: 2024-03-18 15:57:33.68184674 +0000 UTC m=+0.476605516 container start 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T15:57:33.962 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 ceph-mds[67312]: -- 172.21.15.191:0/4080314562 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56476074fa40 con 0x564760723000 2024-03-18T15:57:33.962 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: starting mds.f at 2024-03-18T15:57:33.963 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 bash[67293]: 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 2024-03-18T15:57:33.963 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 15:57:33 smithi191 systemd[1]: Started Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.993+0000 7f8fc41a9640 1 -- 172.21.15.150:0/3296144293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc100700 msgr2=0x7f8fbc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.993+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/3296144293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc100700 0x7f8fbc100b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009900 tx=0x7f8fb802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- 172.21.15.150:0/3296144293 shutdown_connections 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/3296144293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 0x7f8fbc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/3296144293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/3296144293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc100700 0x7f8fbc100b20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- 172.21.15.150:0/3296144293 >> 172.21.15.150:0/3296144293 conn(0x7f8fbc0fbc70 msgr2=0x7f8fbc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:33.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- 172.21.15.150:0/3296144293 shutdown_connections 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- 172.21.15.150:0/3296144293 wait complete. 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 Processor -- start 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- start start 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc100700 0x7f8fbc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 0x7f8fbc19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10c380 con 0x7f8fbc1018f0 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10c200 con 0x7f8fbc100700 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc41a9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10c500 con 0x7f8fbc1022b0 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.994+0000 7f8fc1744640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36916/0 (socket says 172.21.15.150:36916) 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 -- 172.21.15.150:0/2677416128 learned_addr learned my addr 172.21.15.150:0/2677416128 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:33.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1f45640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc100700 0x7f8fbc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc2746640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 0x7f8fbc19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 -- 172.21.15.150:0/2677416128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 msgr2=0x7f8fbc19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 0x7f8fbc19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 -- 172.21.15.150:0/2677416128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc100700 msgr2=0x7f8fbc196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc100700 0x7f8fbc196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 -- 172.21.15.150:0/2677416128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc19e640 con 0x7f8fbc1018f0 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc1744640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00d950 tx=0x7f8fac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac017070 con 0x7f8fbc1018f0 2024-03-18T15:57:33.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac009c30 con 0x7f8fbc1018f0 2024-03-18T15:57:34.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc19e930 con 0x7f8fbc1018f0 2024-03-18T15:57:34.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00f6b0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.995+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc19ee80 con 0x7f8fbc1018f0 2024-03-18T15:57:34.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.996+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc107a20 con 0x7f8fbc1018f0 2024-03-18T15:57:34.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.997+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8fac0109d0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.998+0000 7f8fb2ffd640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 0x7f8fa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:34.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.998+0000 7f8fc1f45640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 0x7f8fa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:34.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.998+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8fac0ce1c0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:33.998+0000 7f8fc1f45640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 0x7f8fa80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8000c00 tx=0x7f8fb8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:34.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.002+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8fac009de0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:33 smithi150 ceph-mon[31946]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:34.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.216+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fbc02d1b0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.216+0000 7f8fb2ffd640 1 -- 172.21.15.150:0/2677416128 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5533 (secure 0 0 0) 0x7f8fac09b0b0 con 0x7f8fbc1018f0 2024-03-18T15:57:34.218 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:34.218 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T15:57:34.218 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 7 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 msgr2=0x7f8fa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 0x7f8fa80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8000c00 tx=0x7f8fb8005fd0 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 msgr2=0x7f8fbc196f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00d950 tx=0x7f8fac00de10 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 shutdown_connections 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f8fa80a2800 0x7f8fa80a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fbc1022b0 0x7f8fbc19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fbc1018f0 0x7f8fbc196f60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 --2- 172.21.15.150:0/2677416128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc100700 0x7f8fbc196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:34.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 >> 172.21.15.150:0/2677416128 conn(0x7f8fbc0fbc70 msgr2=0x7f8fbc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:34.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 shutdown_connections 2024-03-18T15:57:34.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:34.218+0000 7f8fc41a9640 1 -- 172.21.15.150:0/2677416128 wait complete. 2024-03-18T15:57:34.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:33 smithi156 ceph-mon[35919]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6834/2169559605,v1:172.21.15.191:6835/2169559605] up:boot 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: fsmap 6 up:standby 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:35.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:35.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:35.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: Deploying daemon mds.c on smithi191 2024-03-18T15:57:35.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:34 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2677416128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:35.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6834/2169559605,v1:172.21.15.191:6835/2169559605] up:boot 2024-03-18T15:57:35.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: fsmap 6 up:standby 2024-03-18T15:57:35.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:57:35.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: Deploying daemon mds.c on smithi191 2024-03-18T15:57:35.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:34 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2677416128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:35.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6834/2169559605,v1:172.21.15.191:6835/2169559605] up:boot 2024-03-18T15:57:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: fsmap 6 up:standby 2024-03-18T15:57:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T15:57:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:35.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:35.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:35.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:35.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: Deploying daemon mds.c on smithi191 2024-03-18T15:57:35.270 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:34 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2677416128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:36.078 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:35 smithi191 ceph-mon[36784]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:35 smithi150 ceph-mon[31946]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:36.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:36.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:35 smithi156 ceph-mon[35919]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 -- 172.21.15.150:0/3784844689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 msgr2=0x7fb6fc1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 --2- 172.21.15.150:0/3784844689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc1046f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009900 tx=0x7fb6ec02bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 -- 172.21.15.150:0/3784844689 shutdown_connections 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 --2- 172.21.15.150:0/3784844689 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc107560 0x7fb6fc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 --2- 172.21.15.150:0/3784844689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc104c30 0x7fb6fc107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 --2- 172.21.15.150:0/3784844689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc1046f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 -- 172.21.15.150:0/3784844689 >> 172.21.15.150:0/3784844689 conn(0x7fb6fc0fbd10 msgr2=0x7fb6fc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 -- 172.21.15.150:0/3784844689 shutdown_connections 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.344+0000 7fb7018d3640 1 -- 172.21.15.150:0/3784844689 wait complete. 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 Processor -- start 2024-03-18T15:57:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 -- start start 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 0x7fb6fc1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 0x7fb6fc19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10d4c0 con 0x7fb6fc102300 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10d340 con 0x7fb6fc104c30 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb7018d3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10d640 con 0x7fb6fc107560 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb6faffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb6faffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36934/0 (socket says 172.21.15.150:36934) 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb6faffd640 1 -- 172.21.15.150:0/287659968 learned_addr learned my addr 172.21.15.150:0/287659968 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb6fb7fe640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 0x7fb6fc19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.345+0000 7fb6f3fff640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 0x7fb6fc1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 -- 172.21.15.150:0/287659968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 msgr2=0x7fb6fc19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 0x7fb6fc19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 -- 172.21.15.150:0/287659968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 msgr2=0x7fb6fc1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 0x7fb6fc1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 -- 172.21.15.150:0/287659968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc19f790 con 0x7fb6fc102300 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6fb7fe640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 0x7fb6fc19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6f3fff640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 0x7fb6fc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb6faffd640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009230 tx=0x7fb6ec005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec041070 con 0x7fb6fc102300 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc19fa20 con 0x7fb6fc102300 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6ec0044d0 con 0x7fb6fc102300 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec034440 con 0x7fb6fc102300 2024-03-18T15:57:36.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.346+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc19ff10 con 0x7fb6fc102300 2024-03-18T15:57:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.347+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc0fd800 con 0x7fb6fc102300 2024-03-18T15:57:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.348+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb6ec044070 con 0x7fb6fc102300 2024-03-18T15:57:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.348+0000 7fb7008d1640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 0x7fb6e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.349+0000 7fb6f3fff640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 0x7fb6e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.349+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb6ec0f02f0 con 0x7fb6fc102300 2024-03-18T15:57:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.349+0000 7fb6f3fff640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 0x7fb6e80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc199080 tx=0x7fb6e4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:36.352 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 podman[67600]: 2024-03-18 15:57:36.350659965 +0000 UTC m=+0.167801952 container create 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T15:57:36.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.353+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6ec0bd830 con 0x7fb6fc102300 2024-03-18T15:57:36.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.566+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6fc02d1b0 con 0x7fb6fc102300 2024-03-18T15:57:36.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.567+0000 7fb7008d1640 1 -- 172.21.15.150:0/287659968 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5533 (secure 0 0 0) 0x7fb6ec0bd130 con 0x7fb6fc102300 2024-03-18T15:57:36.568 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:36.568 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T15:57:36.569 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 7 2024-03-18T15:57:36.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 msgr2=0x7fb6e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:36.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 0x7fb6e80a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc199080 tx=0x7fb6e4005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:36.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 msgr2=0x7fb6fc197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009230 tx=0x7fb6ec005d50 comp rx=0 tx=0).stop 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 shutdown_connections 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb6e80a2800 0x7fb6e80a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb6fc107560 0x7fb6fc19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6fc104c30 0x7fb6fc1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 --2- 172.21.15.150:0/287659968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6fc102300 0x7fb6fc197b70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 >> 172.21.15.150:0/287659968 conn(0x7fb6fc0fbd10 msgr2=0x7fb6fc102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 shutdown_connections 2024-03-18T15:57:36.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:36.569+0000 7fb7018d3640 1 -- 172.21.15.150:0/287659968 wait complete. 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/287659968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:36 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/287659968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 podman[67600]: 2024-03-18 15:57:36.607240005 +0000 UTC m=+0.424382003 container init 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 podman[67600]: 2024-03-18 15:57:36.6102152 +0000 UTC m=+0.427357198 container start 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-mds[67629]: -- 172.21.15.191:0/2182399998 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e928727a40 con 0x55e9286fa800 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: starting mds.c at 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 bash[67600]: 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e 2024-03-18T15:57:36.963 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 15:57:36 smithi191 systemd[1]: Started Ceph mds.c for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/287659968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:37.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:37.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:36 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:37 smithi156 ceph-mon[35919]: Deploying daemon mds.h on smithi156 2024-03-18T15:57:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:37 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:boot 2024-03-18T15:57:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:37 smithi156 ceph-mon[35919]: fsmap 7 up:standby 2024-03-18T15:57:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:37 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:57:38.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:37 smithi156 ceph-mon[35919]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:37 smithi150 ceph-mon[31946]: Deploying daemon mds.h on smithi156 2024-03-18T15:57:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:37 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:boot 2024-03-18T15:57:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:37 smithi150 ceph-mon[31946]: fsmap 7 up:standby 2024-03-18T15:57:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:37 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:57:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:37 smithi150 ceph-mon[31946]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:38.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:37 smithi191 ceph-mon[36784]: Deploying daemon mds.h on smithi156 2024-03-18T15:57:38.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:37 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:boot 2024-03-18T15:57:38.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:37 smithi191 ceph-mon[36784]: fsmap 7 up:standby 2024-03-18T15:57:38.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:37 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T15:57:38.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:37 smithi191 ceph-mon[36784]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:38.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 -- 172.21.15.150:0/2299938475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac100240 msgr2=0x7fc6ac109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/2299938475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac100240 0x7fc6ac109df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009ca0 tx=0x7fc69c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 -- 172.21.15.150:0/2299938475 shutdown_connections 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/2299938475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac100240 0x7fc6ac109df0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/2299938475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac0ff880 0x7fc6ac0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/2299938475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac102e10 0x7fc6ac103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 -- 172.21.15.150:0/2299938475 >> 172.21.15.150:0/2299938475 conn(0x7fc6ac0fb400 msgr2=0x7fc6ac0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 -- 172.21.15.150:0/2299938475 shutdown_connections 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.695+0000 7fc6b34ac640 1 -- 172.21.15.150:0/2299938475 wait complete. 2024-03-18T15:57:38.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 Processor -- start 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 -- start start 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac0ff880 0x7fc6ac197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac100240 0x7fc6ac198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10d510 con 0x7fc6ac102e10 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10d390 con 0x7fc6ac100240 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b34ac640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10d690 con 0x7fc6ac0ff880 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36948/0 (socket says 172.21.15.150:36948) 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b0a47640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac100240 0x7fc6ac198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 -- 172.21.15.150:0/4026693240 learned_addr learned my addr 172.21.15.150:0/4026693240 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1248640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac0ff880 0x7fc6ac197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 -- 172.21.15.150:0/4026693240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac0ff880 msgr2=0x7fc6ac197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:38.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac0ff880 0x7fc6ac197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 -- 172.21.15.150:0/4026693240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac100240 msgr2=0x7fc6ac198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac100240 0x7fc6ac198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 -- 172.21.15.150:0/4026693240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac19f720 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6b1a49640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc69c037040 tx=0x7fc69c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.696+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c041020 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.697+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc69c034d90 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.697+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac19f9b0 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.697+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c039040 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.697+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac19fea0 con 0x7fc6ac102e10 2024-03-18T15:57:38.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.697+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac108cd0 con 0x7fc6ac102e10 2024-03-18T15:57:38.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.699+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc69c044020 con 0x7fc6ac102e10 2024-03-18T15:57:38.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.700+0000 7fc6aa7fc640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 0x7fc6940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:38.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.700+0000 7fc6b1248640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 0x7fc6940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:38.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.700+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc69c0f0a60 con 0x7fc6ac102e10 2024-03-18T15:57:38.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.700+0000 7fc6b1248640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 0x7fc6940a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac069b10 tx=0x7fc6a4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:38.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.703+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc69c04b020 con 0x7fc6ac102e10 2024-03-18T15:57:38.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.916+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6ac02d1b0 con 0x7fc6ac102e10 2024-03-18T15:57:38.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.916+0000 7fc6aa7fc640 1 -- 172.21.15.150:0/4026693240 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fc69c0bdfa0 con 0x7fc6ac102e10 2024-03-18T15:57:38.918 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:38.918 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T15:57:38.918 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 msgr2=0x7fc6940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 0x7fc6940a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac069b10 tx=0x7fc6a4008040 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 msgr2=0x7fc6ac19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc69c037040 tx=0x7fc69c032f20 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 shutdown_connections 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc6940a2800 0x7fc6940a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6ac102e10 0x7fc6ac19f0b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6ac100240 0x7fc6ac198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 --2- 172.21.15.150:0/4026693240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6ac0ff880 0x7fc6ac197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:38.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.918+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 >> 172.21.15.150:0/4026693240 conn(0x7fc6ac0fb400 msgr2=0x7fc6ac105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:38.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.919+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 shutdown_connections 2024-03-18T15:57:38.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:38.919+0000 7fc6b34ac640 1 -- 172.21.15.150:0/4026693240 wait complete. 2024-03-18T15:57:39.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:38 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4026693240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:39.250 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 systemd[1]: Starting Ceph mds.h for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:39.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:38 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4026693240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:39.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:38 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4026693240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:39.518 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 podman[66031]: 2024-03-18 15:57:39.152311379 +0000 UTC m=+0.018099287 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:39.518 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 podman[66031]: 2024-03-18 15:57:39.280998457 +0000 UTC m=+0.146786350 container create 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T15:57:39.971 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 podman[66031]: 2024-03-18 15:57:39.525740274 +0000 UTC m=+0.391528192 container init 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:39.971 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 podman[66031]: 2024-03-18 15:57:39.52837394 +0000 UTC m=+0.394161834 container start 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T15:57:39.971 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mds[66049]: -- 172.21.15.156:0/3545167704 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d73387a40 con 0x560d7335ac00 2024-03-18T15:57:39.971 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: starting mds.h at 2024-03-18T15:57:39.971 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 bash[66031]: 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 2024-03-18T15:57:39.972 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 15:57:39 smithi156 systemd[1]: Started Ceph mds.h for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:40.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:40.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:40.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:40.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:39 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:40.332 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:40.333 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:39 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:40.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:39 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:40.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1165620391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4101910 msgr2=0x7f66e4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1165620391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4101910 0x7f66e4101d90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009900 tx=0x7f66d402bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1165620391 shutdown_connections 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1165620391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e41022d0 0x7f66e4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1165620391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4101910 0x7f66e4101d90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1165620391 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e4100720 0x7f66e4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1165620391 >> 172.21.15.150:0/1165620391 conn(0x7f66e40fbc90 msgr2=0x7f66e40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1165620391 shutdown_connections 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1165620391 wait complete. 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 Processor -- start 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.043+0000 7f66ec7f3640 1 -- start start 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 0x7f66e41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e41022d0 0x7f66e419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410c3a0 con 0x7f66e4100720 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410c220 con 0x7f66e4101910 2024-03-18T15:57:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f66e410c520 con 0x7f66e41022d0 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36964/0 (socket says 172.21.15.150:36964) 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 -- 172.21.15.150:0/1786362038 learned_addr learned my addr 172.21.15.150:0/1786362038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ead90640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e41022d0 0x7f66e419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66e9d8e640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 0x7f66e41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 -- 172.21.15.150:0/1786362038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e41022d0 msgr2=0x7f66e419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e41022d0 0x7f66e419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 -- 172.21.15.150:0/1786362038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 msgr2=0x7f66e41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 0x7f66e41980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 -- 172.21.15.150:0/1786362038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e419f790 con 0x7f66e4100720 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66e9d8e640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 0x7f66e41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ea58f640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d950 tx=0x7f66e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0017070 con 0x7f66e4100720 2024-03-18T15:57:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e0009c30 con 0x7f66e4100720 2024-03-18T15:57:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e419fa80 con 0x7f66e4100720 2024-03-18T15:57:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e000f600 con 0x7f66e4100720 2024-03-18T15:57:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.044+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e419ff80 con 0x7f66e4100720 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.046+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e41a0150 con 0x7f66e4100720 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.047+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f66e001a070 con 0x7f66e4100720 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.047+0000 7f66db7fe640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 0x7f66d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.048+0000 7f66e9d8e640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 0x7f66d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.048+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f66e00ce020 con 0x7f66e4100720 2024-03-18T15:57:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.048+0000 7f66e9d8e640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 0x7f66d00a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f66d4036040 tx=0x7f66d4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.052+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66e0009de0 con 0x7f66e4100720 2024-03-18T15:57:41.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:40 smithi156 ceph-mon[35919]: Deploying daemon mds.i on smithi191 2024-03-18T15:57:41.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:41 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6836/1894652569,v1:172.21.15.156:6837/1894652569] up:boot 2024-03-18T15:57:41.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:41 smithi156 ceph-mon[35919]: fsmap 8 up:standby 2024-03-18T15:57:41.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:41 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:57:41.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.268+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66e41a0440 con 0x7f66e4100720 2024-03-18T15:57:41.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.268+0000 7f66db7fe640 1 -- 172.21.15.150:0/1786362038 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7f66e009b560 con 0x7f66e4100720 2024-03-18T15:57:41.269 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:41.269 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T15:57:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 9 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.270+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 msgr2=0x7f66d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.270+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 0x7f66d00a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f66d4036040 tx=0x7f66d4000bc0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.270+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 msgr2=0x7f66e4197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.270+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d950 tx=0x7f66e000de10 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 shutdown_connections 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66d00a2800 0x7f66d00a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e41022d0 0x7f66e419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66e4101910 0x7f66e41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 --2- 172.21.15.150:0/1786362038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66e4100720 0x7f66e4197b70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 >> 172.21.15.150:0/1786362038 conn(0x7f66e40fbc90 msgr2=0x7f66e40fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:41.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 shutdown_connections 2024-03-18T15:57:41.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:41.271+0000 7f66ec7f3640 1 -- 172.21.15.150:0/1786362038 wait complete. 2024-03-18T15:57:41.281 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:40 smithi150 ceph-mon[31946]: Deploying daemon mds.i on smithi191 2024-03-18T15:57:41.281 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:40 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6836/1894652569,v1:172.21.15.156:6837/1894652569] up:boot 2024-03-18T15:57:41.281 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:40 smithi150 ceph-mon[31946]: fsmap 8 up:standby 2024-03-18T15:57:41.281 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:40 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:57:41.338 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:40 smithi191 ceph-mon[36784]: Deploying daemon mds.i on smithi191 2024-03-18T15:57:41.338 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:40 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6836/1894652569,v1:172.21.15.156:6837/1894652569] up:boot 2024-03-18T15:57:41.338 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:40 smithi191 ceph-mon[36784]: fsmap 8 up:standby 2024-03-18T15:57:41.338 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:40 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T15:57:41.989 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:41 smithi191 systemd[1]: Starting Ceph mds.i for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:42.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:41 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1786362038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:42.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:41 smithi156 ceph-mon[35919]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:42.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:41 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1786362038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:42.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:41 smithi191 ceph-mon[36784]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:42.275 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 podman[67921]: 2024-03-18 15:57:42.038261373 +0000 UTC m=+0.017564201 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:42.275 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 podman[67921]: 2024-03-18 15:57:42.157153374 +0000 UTC m=+0.136456233 container create ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:57:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:41 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1786362038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:42.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:41 smithi150 ceph-mon[31946]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:42.546 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 podman[67921]: 2024-03-18 15:57:42.428287207 +0000 UTC m=+0.407590014 container init ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T15:57:42.546 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 podman[67921]: 2024-03-18 15:57:42.431126493 +0000 UTC m=+0.410429313 container start ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:42.546 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 ceph-mds[67939]: -- 172.21.15.191:0/812171802 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0507b1a40 con 0x55e050784800 2024-03-18T15:57:42.546 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: starting mds.i at 2024-03-18T15:57:42.962 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 bash[67921]: ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb 2024-03-18T15:57:42.962 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:57:42 smithi191 systemd[1]: Started Ceph mds.i for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:43.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- 172.21.15.150:0/3199118210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550107520 msgr2=0x7fc550109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 --2- 172.21.15.150:0/3199118210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550107520 0x7fc550109930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc548009ca0 tx=0x7fc54802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- 172.21.15.150:0/3199118210 shutdown_connections 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 --2- 172.21.15.150:0/3199118210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550107520 0x7fc550109930 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 --2- 172.21.15.150:0/3199118210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc550104bf0 0x7fc550106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 --2- 172.21.15.150:0/3199118210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5501022c0 0x7fc5501046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- 172.21.15.150:0/3199118210 >> 172.21.15.150:0/3199118210 conn(0x7fc5500fbc70 msgr2=0x7fc5500fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- 172.21.15.150:0/3199118210 shutdown_connections 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- 172.21.15.150:0/3199118210 wait complete. 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 Processor -- start 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.388+0000 7fc555bb9640 1 -- start start 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5501022c0 0x7fc550197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc550107520 0x7fc55019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010d4a0 con 0x7fc550104bf0 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010d320 con 0x7fc5501022c0 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc55010d620 con 0x7fc550107520 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:43.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60022/0 (socket says 172.21.15.150:60022) 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 -- 172.21.15.150:0/1063840702 learned_addr learned my addr 172.21.15.150:0/1063840702 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54ffff640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc550107520 0x7fc55019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 -- 172.21.15.150:0/1063840702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc550107520 msgr2=0x7fc55019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc550107520 0x7fc55019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54f7fe640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5501022c0 0x7fc550197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 -- 172.21.15.150:0/1063840702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5501022c0 msgr2=0x7fc550197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5501022c0 0x7fc550197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 -- 172.21.15.150:0/1063840702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55019f710 con 0x7fc550104bf0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54effd640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc54400d770 tx=0x7fc54400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544017070 con 0x7fc550104bf0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55019fa00 con 0x7fc550104bf0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc544009c30 con 0x7fc550104bf0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54400f620 con 0x7fc550104bf0 2024-03-18T15:57:43.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.389+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55019ff50 con 0x7fc550104bf0 2024-03-18T15:57:43.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.390+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5500fd760 con 0x7fc550104bf0 2024-03-18T15:57:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.391+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc54401a070 con 0x7fc550104bf0 2024-03-18T15:57:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.392+0000 7fc54cff9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 0x7fc5380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.392+0000 7fc54f7fe640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 0x7fc5380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.392+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5440ce140 con 0x7fc550104bf0 2024-03-18T15:57:43.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.392+0000 7fc54f7fe640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 0x7fc5380a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc540005fd0 tx=0x7fc540005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:43.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.397+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc544009de0 con 0x7fc550104bf0 2024-03-18T15:57:43.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.623+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc55002d3b0 con 0x7fc550104bf0 2024-03-18T15:57:43.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.624+0000 7fc54cff9640 1 -- 172.21.15.150:0/1063840702 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7fc54409b680 con 0x7fc550104bf0 2024-03-18T15:57:43.625 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:43.625 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:57:43.626 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 10 2024-03-18T15:57:43.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 msgr2=0x7fc5380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:43.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 0x7fc5380a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc540005fd0 tx=0x7fc540005850 comp rx=0 tx=0).stop 2024-03-18T15:57:43.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 msgr2=0x7fc550198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc54400d770 tx=0x7fc54400dc30 comp rx=0 tx=0).stop 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 shutdown_connections 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc5380a2800 0x7fc5380a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc550107520 0x7fc55019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc550104bf0 0x7fc550198030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 --2- 172.21.15.150:0/1063840702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5501022c0 0x7fc550197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 >> 172.21.15.150:0/1063840702 conn(0x7fc5500fbc70 msgr2=0x7fc550102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 shutdown_connections 2024-03-18T15:57:43.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:43.626+0000 7fc555bb9640 1 -- 172.21.15.150:0/1063840702 wait complete. 2024-03-18T15:57:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:boot 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: fsmap 9 up:standby 2024-03-18T15:57:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:43 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:57:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:boot 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: fsmap 9 up:standby 2024-03-18T15:57:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:43 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:57:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:44.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:44.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:44.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:boot 2024-03-18T15:57:44.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: fsmap 9 up:standby 2024-03-18T15:57:44.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:43 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T15:57:44.861 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:44 smithi150 systemd[1]: Starting Ceph mds.g for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:44.862 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:44 smithi150 ceph-mon[31946]: Deploying daemon mds.g on smithi150 2024-03-18T15:57:44.862 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:44 smithi150 ceph-mon[31946]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:44.862 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:44 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1063840702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:44 smithi191 ceph-mon[36784]: Deploying daemon mds.g on smithi150 2024-03-18T15:57:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:44 smithi191 ceph-mon[36784]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:44 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1063840702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:45.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:44 smithi156 ceph-mon[35919]: Deploying daemon mds.g on smithi150 2024-03-18T15:57:45.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:44 smithi156 ceph-mon[35919]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:45.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:44 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1063840702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:45.233 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 podman[71069]: 2024-03-18 15:57:44.980012544 +0000 UTC m=+0.017716978 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:45.233 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 podman[71069]: 2024-03-18 15:57:45.105909252 +0000 UTC m=+0.143613728 container create ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 podman[71069]: 2024-03-18 15:57:45.375844523 +0000 UTC m=+0.413548968 container init ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 podman[71069]: 2024-03-18 15:57:45.378513361 +0000 UTC m=+0.416217804 container start ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 ceph-mds[71088]: -- 172.21.15.150:0/337327080 <== mon.0 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd3a83fa40 con 0x55fd3a812c00 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g[71084]: starting mds.g at 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 bash[71069]: ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 2024-03-18T15:57:45.592 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 15:57:45 smithi150 systemd[1]: Started Ceph mds.g for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:45.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:45.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 -- 172.21.15.150:0/894533644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681022b0 msgr2=0x7fae68108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 --2- 172.21.15.150:0/894533644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681022b0 0x7fae68108b40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae60009ca0 tx=0x7fae6002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 -- 172.21.15.150:0/894533644 shutdown_connections 2024-03-18T15:57:45.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 --2- 172.21.15.150:0/894533644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681022b0 0x7fae68108b40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 --2- 172.21.15.150:0/894533644 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681018f0 0x7fae68101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 --2- 172.21.15.150:0/894533644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 -- 172.21.15.150:0/894533644 >> 172.21.15.150:0/894533644 conn(0x7fae680fbc70 msgr2=0x7fae680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 -- 172.21.15.150:0/894533644 shutdown_connections 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 -- 172.21.15.150:0/894533644 wait complete. 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.746+0000 7fae6efa9640 1 Processor -- start 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 -- start start 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 0x7fae68103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810c360 con 0x7fae681018f0 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810c1e0 con 0x7fae68100700 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fae6810c4e0 con 0x7fae681022b0 2024-03-18T15:57:45.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60060/0 (socket says 172.21.15.150:60060) 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 -- 172.21.15.150:0/1097129091 learned_addr learned my addr 172.21.15.150:0/1097129091 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6cd45640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6d546640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 0x7fae68103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 -- 172.21.15.150:0/1097129091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 msgr2=0x7fae68103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 0x7fae68103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 -- 172.21.15.150:0/1097129091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 msgr2=0x7fae68104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 -- 172.21.15.150:0/1097129091 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae68103aa0 con 0x7fae681018f0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6cd45640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6d546640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 0x7fae68103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae67fff640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae5400c920 tx=0x7fae5400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54007c70 con 0x7fae681018f0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae54007e10 con 0x7fae681018f0 2024-03-18T15:57:45.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae681a1b80 con 0x7fae681018f0 2024-03-18T15:57:45.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.747+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54018890 con 0x7fae681018f0 2024-03-18T15:57:45.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.748+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae681a2120 con 0x7fae681018f0 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.749+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae681a2350 con 0x7fae681018f0 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.750+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fae54018a70 con 0x7fae681018f0 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.750+0000 7fae65ffb640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 0x7fae580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.750+0000 7fae6cd45640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 0x7fae580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.750+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fae540ce7b0 con 0x7fae681018f0 2024-03-18T15:57:45.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.751+0000 7fae6cd45640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 0x7fae580a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0058c0 tx=0x7fae5c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:45.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.755+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae54014030 con 0x7fae681018f0 2024-03-18T15:57:45.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.977+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae681a2520 con 0x7fae681018f0 2024-03-18T15:57:45.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.977+0000 7fae65ffb640 1 -- 172.21.15.150:0/1097129091 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7fae5400de20 con 0x7fae681018f0 2024-03-18T15:57:45.980 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:45.980 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:57:45.980 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 11 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.979+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 msgr2=0x7fae580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.979+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 0x7fae580a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0058c0 tx=0x7fae5c005850 comp rx=0 tx=0).stop 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 msgr2=0x7fae68102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae5400c920 tx=0x7fae5400cde0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 shutdown_connections 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae580a2880 0x7fae580a4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae681022b0 0x7fae68103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae681018f0 0x7fae68102cd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 --2- 172.21.15.150:0/1097129091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae68100700 0x7fae68104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 >> 172.21.15.150:0/1097129091 conn(0x7fae680fbc70 msgr2=0x7fae680fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:45.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 shutdown_connections 2024-03-18T15:57:45.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:45.980+0000 7fae6efa9640 1 -- 172.21.15.150:0/1097129091 wait complete. 2024-03-18T15:57:46.626 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:46.626 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.626 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.626 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.626 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: Deploying daemon mds.a on smithi150 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.150:6838/4258774828,v1:172.21.15.150:6839/4258774828] up:boot 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: fsmap 10 up:standby 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:57:46.627 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1097129091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: Deploying daemon mds.a on smithi150 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.150:6838/4258774828,v1:172.21.15.150:6839/4258774828] up:boot 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: fsmap 10 up:standby 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:57:46.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:46 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1097129091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:47.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: Deploying daemon mds.a on smithi150 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.150:6838/4258774828,v1:172.21.15.150:6839/4258774828] up:boot 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: fsmap 10 up:standby 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T15:57:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:46 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1097129091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:47.842 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:47 smithi150 systemd[1]: Starting Ceph mds.a for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:47.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:48.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 -- 172.21.15.150:0/3338146651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 msgr2=0x7f0034108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 --2- 172.21.15.150:0/3338146651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f0034108b40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f002c009900 tx=0x7f002c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 -- 172.21.15.150:0/3338146651 shutdown_connections 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 --2- 172.21.15.150:0/3338146651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f0034108b40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 --2- 172.21.15.150:0/3338146651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00341018f0 0x7f0034101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 --2- 172.21.15.150:0/3338146651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0034100700 0x7f0034100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 -- 172.21.15.150:0/3338146651 >> 172.21.15.150:0/3338146651 conn(0x7f00340fbc70 msgr2=0x7f00340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 -- 172.21.15.150:0/3338146651 shutdown_connections 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 -- 172.21.15.150:0/3338146651 wait complete. 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.110+0000 7f003a176640 1 Processor -- start 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 -- start start 2024-03-18T15:57:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0034100700 0x7f00340694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00341018f0 0x7f003419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c3e0 con 0x7f00341022b0 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c260 con 0x7f00341018f0 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f003a176640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c560 con 0x7f0034100700 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60080/0 (socket says 172.21.15.150:60080) 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 -- 172.21.15.150:0/769438929 learned_addr learned my addr 172.21.15.150:0/769438929 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0032ffd640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00341018f0 0x7f003419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f00337fe640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0034100700 0x7f00340694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 -- 172.21.15.150:0/769438929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0034100700 msgr2=0x7f00340694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0034100700 0x7f00340694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 -- 172.21.15.150:0/769438929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00341018f0 msgr2=0x7f003419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00341018f0 0x7f003419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 -- 172.21.15.150:0/769438929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003419f7a0 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.111+0000 7f0033fff640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f002c02ce70 tx=0x7f002c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.112+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c041070 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.112+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f002c0054e0 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.112+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003419f9d0 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.112+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c034440 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.112+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003419ff10 con 0x7f00341022b0 2024-03-18T15:57:48.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.113+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00341079c0 con 0x7f00341022b0 2024-03-18T15:57:48.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.114+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f002c044070 con 0x7f00341022b0 2024-03-18T15:57:48.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.114+0000 7f0030ff9640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 0x7f00200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.114+0000 7f00337fe640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 0x7f00200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.114+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f002c0f0700 con 0x7f00341022b0 2024-03-18T15:57:48.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.115+0000 7f00337fe640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 0x7f00200a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0034101750 tx=0x7f0024009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:48.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.118+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f002c0bdc40 con 0x7f00341022b0 2024-03-18T15:57:48.192 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:47 smithi150 podman[71426]: 2024-03-18 15:57:47.842411728 +0000 UTC m=+0.018828096 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:48.192 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:47 smithi150 podman[71426]: 2024-03-18 15:57:47.954693329 +0000 UTC m=+0.131109719 container create 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T15:57:48.192 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 podman[71426]: 2024-03-18 15:57:48.191511136 +0000 UTC m=+0.367927496 container init 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T15:57:48.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.338+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f003402d1b0 con 0x7f00341022b0 2024-03-18T15:57:48.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.341+0000 7f0030ff9640 1 -- 172.21.15.150:0/769438929 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7f002c0bd540 con 0x7f00341022b0 2024-03-18T15:57:48.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:48.343 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:57:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 11 2024-03-18T15:57:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 msgr2=0x7f00200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 0x7f00200a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0034101750 tx=0x7f0024009870 comp rx=0 tx=0).stop 2024-03-18T15:57:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 msgr2=0x7f003419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f002c02ce70 tx=0x7f002c005e40 comp rx=0 tx=0).stop 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 shutdown_connections 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00200a2880 0x7f00200a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00341022b0 0x7f003419ee90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00341018f0 0x7f003419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 --2- 172.21.15.150:0/769438929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0034100700 0x7f00340694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 >> 172.21.15.150:0/769438929 conn(0x7f00340fbc70 msgr2=0x7f0034104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 shutdown_connections 2024-03-18T15:57:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:48.343+0000 7f003a176640 1 -- 172.21.15.150:0/769438929 wait complete. 2024-03-18T15:57:48.535 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 podman[71426]: 2024-03-18 15:57:48.194722017 +0000 UTC m=+0.371138373 container start 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T15:57:48.535 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mds[71484]: -- 172.21.15.150:0/1163888723 <== mon.0 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dfd39afa40 con 0x55dfd3983000 2024-03-18T15:57:48.535 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a[71461]: starting mds.a at 2024-03-18T15:57:48.535 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 bash[71426]: 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 2024-03-18T15:57:48.535 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 15:57:48 smithi150 systemd[1]: Started Ceph mds.a for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/769438929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:48.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:48 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/769438929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:48.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:48 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/769438929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:49.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:49.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:49.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T15:57:49.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T15:57:49.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:48 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:49.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:49 smithi150 ceph-mon[31946]: Deploying daemon mds.b on smithi156 2024-03-18T15:57:49.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:49 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.150:6840/4215738622,v1:172.21.15.150:6841/4215738622] up:boot 2024-03-18T15:57:49.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:49 smithi150 ceph-mon[31946]: fsmap 11 up:standby 2024-03-18T15:57:49.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:49 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:57:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:49 smithi191 ceph-mon[36784]: Deploying daemon mds.b on smithi156 2024-03-18T15:57:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:49 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.150:6840/4215738622,v1:172.21.15.150:6841/4215738622] up:boot 2024-03-18T15:57:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:49 smithi191 ceph-mon[36784]: fsmap 11 up:standby 2024-03-18T15:57:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:49 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:57:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:49 smithi156 ceph-mon[35919]: Deploying daemon mds.b on smithi156 2024-03-18T15:57:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:49 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.150:6840/4215738622,v1:172.21.15.150:6841/4215738622] up:boot 2024-03-18T15:57:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:49 smithi156 ceph-mon[35919]: fsmap 11 up:standby 2024-03-18T15:57:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:49 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T15:57:50.357 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 -- 172.21.15.150:0/1940644387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc107370 msgr2=0x7f00dc109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 --2- 172.21.15.150:0/1940644387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc107370 0x7f00dc109760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009ca0 tx=0x7f00d402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 -- 172.21.15.150:0/1940644387 shutdown_connections 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 --2- 172.21.15.150:0/1940644387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc107370 0x7f00dc109760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 --2- 172.21.15.150:0/1940644387 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 0x7f00dc106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 --2- 172.21.15.150:0/1940644387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc102110 0x7f00dc104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 -- 172.21.15.150:0/1940644387 >> 172.21.15.150:0/1940644387 conn(0x7f00dc0fbac0 msgr2=0x7f00dc0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 -- 172.21.15.150:0/1940644387 shutdown_connections 2024-03-18T15:57:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.470+0000 7f00e139d640 1 -- 172.21.15.150:0/1940644387 wait complete. 2024-03-18T15:57:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 Processor -- start 2024-03-18T15:57:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 -- start start 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 0x7f00dc197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc107370 0x7f00dc19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc10d2c0 con 0x7f00dc102110 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc10d140 con 0x7f00dc107370 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.471+0000 7f00e139d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc10d440 con 0x7f00dc104a40 2024-03-18T15:57:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60126/0 (socket says 172.21.15.150:60126) 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 -- 172.21.15.150:0/34103120 learned_addr learned my addr 172.21.15.150:0/34103120 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00e0b9c640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc107370 0x7f00dc19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00db7fe640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 0x7f00dc197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 -- 172.21.15.150:0/34103120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 msgr2=0x7f00dc197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 0x7f00dc197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 -- 172.21.15.150:0/34103120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc107370 msgr2=0x7f00dc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc107370 0x7f00dc19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 -- 172.21.15.150:0/34103120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc19f520 con 0x7f00dc102110 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00dbfff640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00e9f0 tx=0x7f00cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00cd80 con 0x7f00dc102110 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00cc010910 con 0x7f00dc102110 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc19f810 con 0x7f00dc102110 2024-03-18T15:57:50.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc018990 con 0x7f00dc102110 2024-03-18T15:57:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.472+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc19fdb0 con 0x7f00dc102110 2024-03-18T15:57:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.473+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc0fd5b0 con 0x7f00dc102110 2024-03-18T15:57:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.474+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f00cc010ac0 con 0x7f00dc102110 2024-03-18T15:57:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.475+0000 7f00d97fa640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 0x7f00c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:50.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.475+0000 7f00db7fe640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 0x7f00c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:50.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.475+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f00cc0ced00 con 0x7f00dc102110 2024-03-18T15:57:50.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.475+0000 7f00db7fe640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 0x7f00c40a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f00d0005fd0 tx=0x7f00d0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:50.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.478+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00cc09c240 con 0x7f00dc102110 2024-03-18T15:57:50.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.691+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00dc02d400 con 0x7f00dc102110 2024-03-18T15:57:50.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.692+0000 7f00d97fa640 1 -- 172.21.15.150:0/34103120 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9674 (secure 0 0 0) 0x7f00cc09bb40 con 0x7f00dc102110 2024-03-18T15:57:50.694 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:50.694 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:57:50.694 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 12 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 msgr2=0x7f00c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 0x7f00c40a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f00d0005fd0 tx=0x7f00d0005850 comp rx=0 tx=0).stop 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 msgr2=0x7f00dc197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00e9f0 tx=0x7f00cc00eeb0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 shutdown_connections 2024-03-18T15:57:50.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f00c40a2880 0x7f00c40a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00dc107370 0x7f00dc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00dc104a40 0x7f00dc197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 --2- 172.21.15.150:0/34103120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc102110 0x7f00dc197900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 >> 172.21.15.150:0/34103120 conn(0x7f00dc0fbac0 msgr2=0x7f00dc1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 shutdown_connections 2024-03-18T15:57:50.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:50.694+0000 7f00e139d640 1 -- 172.21.15.150:0/34103120 wait complete. 2024-03-18T15:57:50.706 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:50 smithi150 ceph-mon[31946]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:50.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:50 smithi156 ceph-mon[35919]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:50.768 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:50 smithi156 systemd[1]: Starting Ceph mds.b for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T15:57:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:50 smithi191 ceph-mon[36784]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:51.123 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:50 smithi156 podman[66352]: 2024-03-18 15:57:50.771227772 +0000 UTC m=+0.018030977 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T15:57:51.123 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:50 smithi156 podman[66352]: 2024-03-18 15:57:50.8782107 +0000 UTC m=+0.125013895 container create 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:57:51.124 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 podman[66352]: 2024-03-18 15:57:51.122742207 +0000 UTC m=+0.369545422 container init 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T15:57:51.518 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 podman[66352]: 2024-03-18 15:57:51.125365157 +0000 UTC m=+0.372168364 container start 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T15:57:51.518 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mds[66371]: -- 172.21.15.156:0/366164796 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55635d60ba40 con 0x55635d5dec00 2024-03-18T15:57:51.518 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: starting mds.b at 2024-03-18T15:57:51.518 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 bash[66352]: 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 2024-03-18T15:57:51.519 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 15:57:51 smithi156 systemd[1]: Started Ceph mds.b for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/34103120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:51 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/34103120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:51 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/34103120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:52.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:52.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:51 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:52.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 -- 172.21.15.150:0/1335519426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1281008f0 msgr2=0x7fe128100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 --2- 172.21.15.150:0/1335519426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1281008f0 0x7fe128100d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe118009900 tx=0x7fe11802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 -- 172.21.15.150:0/1335519426 shutdown_connections 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 --2- 172.21.15.150:0/1335519426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101290 0x7fe128108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 --2- 172.21.15.150:0/1335519426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1281008f0 0x7fe128100d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 --2- 172.21.15.150:0/1335519426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe128101ca0 0x7fe1281020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.817+0000 7fe130226640 1 -- 172.21.15.150:0/1335519426 >> 172.21.15.150:0/1335519426 conn(0x7fe1280fb3e0 msgr2=0x7fe1280fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- 172.21.15.150:0/1335519426 shutdown_connections 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- 172.21.15.150:0/1335519426 wait complete. 2024-03-18T15:57:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 Processor -- start 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- start start 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 0x7fe128100370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 0x7fe12819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe12810c330 con 0x7fe128101290 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe12810c1b0 con 0x7fe128101ca0 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe130226640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe12810c4b0 con 0x7fe1281008f0 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe12d7c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe12d7c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60604/0 (socket says 172.21.15.150:60604) 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe12e7c3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 0x7fe12819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe12d7c1640 1 -- 172.21.15.150:0/1940686425 learned_addr learned my addr 172.21.15.150:0/1940686425 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.818+0000 7fe12dfc2640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 0x7fe128100370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 -- 172.21.15.150:0/1940686425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 msgr2=0x7fe12819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 0x7fe12819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 -- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 msgr2=0x7fe12819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 -- 172.21.15.150:0/1940686425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12819f6d0 con 0x7fe1281008f0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12e7c3640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 0x7fe12819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12d7c1640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe12dfc2640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 0x7fe128100370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe12400e9f0 tx=0x7fe12400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12400ccc0 con 0x7fe1281008f0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12819f9c0 con 0x7fe1281008f0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe124010910 con 0x7fe1281008f0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe124018960 con 0x7fe1281008f0 2024-03-18T15:57:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.819+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12819fee0 con 0x7fe1281008f0 2024-03-18T15:57:52.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.820+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe128107a30 con 0x7fe1281008f0 2024-03-18T15:57:52.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.822+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe124010ac0 con 0x7fe1281008f0 2024-03-18T15:57:52.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.822+0000 7fe11effd640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 0x7fe1140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:52.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.822+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe1240cee60 con 0x7fe1281008f0 2024-03-18T15:57:52.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.822+0000 7fe12d7c1640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 0x7fe1140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:52.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.823+0000 7fe12d7c1640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 0x7fe1140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c00 tx=0x7fe11802c880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:52.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:52.826+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe12409bd50 con 0x7fe1281008f0 2024-03-18T15:57:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:52 smithi150 ceph-mon[31946]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:52 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:boot 2024-03-18T15:57:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:52 smithi150 ceph-mon[31946]: fsmap 12 up:standby 2024-03-18T15:57:52.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:52 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:57:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:52 smithi191 ceph-mon[36784]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:52 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:boot 2024-03-18T15:57:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:52 smithi191 ceph-mon[36784]: fsmap 12 up:standby 2024-03-18T15:57:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:52 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:57:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:52 smithi156 ceph-mon[35919]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:52 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:boot 2024-03-18T15:57:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:52 smithi156 ceph-mon[35919]: fsmap 12 up:standby 2024-03-18T15:57:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:52 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T15:57:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.070+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe12802d370 con 0x7fe1281008f0 2024-03-18T15:57:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.071+0000 7fe11effd640 1 -- 172.21.15.150:0/1940686425 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7fe12409fdf0 con 0x7fe1281008f0 2024-03-18T15:57:53.072 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:53.072 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24485,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T15:57:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 13 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 msgr2=0x7fe1140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 0x7fe1140a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c00 tx=0x7fe11802c880 comp rx=0 tx=0).stop 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 msgr2=0x7fe128100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 0x7fe128100370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe12400e9f0 tx=0x7fe12400eeb0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 shutdown_connections 2024-03-18T15:57:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1140a2800 0x7fe1140a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe128101ca0 0x7fe12819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe128101290 0x7fe12819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 --2- 172.21.15.150:0/1940686425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1281008f0 0x7fe128100370 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 >> 172.21.15.150:0/1940686425 conn(0x7fe1280fb3e0 msgr2=0x7fe128104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 shutdown_connections 2024-03-18T15:57:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.073+0000 7fe130226640 1 -- 172.21.15.150:0/1940686425 wait complete. 2024-03-18T15:57:53.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 -- 172.21.15.150:0/1635973224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6228100700 msgr2=0x7f6228100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 --2- 172.21.15.150:0/1635973224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6228100700 0x7f6228100b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6218009900 tx=0x7f621802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 -- 172.21.15.150:0/1635973224 shutdown_connections 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 --2- 172.21.15.150:0/1635973224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62281022b0 0x7f6228108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 --2- 172.21.15.150:0/1635973224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 0x7f6228101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 --2- 172.21.15.150:0/1635973224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6228100700 0x7f6228100b20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 -- 172.21.15.150:0/1635973224 >> 172.21.15.150:0/1635973224 conn(0x7f62280fbc70 msgr2=0x7f62280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 -- 172.21.15.150:0/1635973224 shutdown_connections 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 -- 172.21.15.150:0/1635973224 wait complete. 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.191+0000 7f622e7cd640 1 Processor -- start 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 -- start start 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6228100700 0x7f6228197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 0x7f6228198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f622810d4a0 con 0x7f62281022b0 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f622810d320 con 0x7f62281018f0 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622e7cd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f622810d620 con 0x7f6228100700 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f6227fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6228100700 0x7f6228197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60616/0 (socket says 172.21.15.150:60616) 2024-03-18T15:57:53.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 -- 172.21.15.150:0/3198323282 learned_addr learned my addr 172.21.15.150:0/3198323282 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f62277fe640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 0x7f6228198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 -- 172.21.15.150:0/3198323282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6228100700 msgr2=0x7f6228197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6228100700 0x7f6228197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 -- 172.21.15.150:0/3198323282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 msgr2=0x7f6228198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 0x7f6228198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 -- 172.21.15.150:0/3198323282 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622819f6f0 con 0x7f62281022b0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.192+0000 7f622cd6a640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f622000cad0 tx=0x7f622000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.193+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622000ad60 con 0x7f62281022b0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.193+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6220005020 con 0x7f62281022b0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.193+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622819f9e0 con 0x7f62281022b0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.193+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622000f6b0 con 0x7f62281022b0 2024-03-18T15:57:53.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.193+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622819fee0 con 0x7f62281022b0 2024-03-18T15:57:53.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.194+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6228107a20 con 0x7f62281022b0 2024-03-18T15:57:53.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.195+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6220005560 con 0x7f62281022b0 2024-03-18T15:57:53.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.195+0000 7f62257fa640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 0x7f62100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.195+0000 7f6227fff640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 0x7f62100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.196+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f62200ce8f0 con 0x7f62281022b0 2024-03-18T15:57:53.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.196+0000 7f6227fff640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 0x7f62100a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6218000c00 tx=0x7f6218004f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.199+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6220014030 con 0x7f62281022b0 2024-03-18T15:57:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.386+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f622802d330 con 0x7f62281022b0 2024-03-18T15:57:53.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.387+0000 7f62257fa640 1 -- 172.21.15.150:0/3198323282 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18512 (secure 0 0 0) 0x7f622009be30 con 0x7f62281022b0 2024-03-18T15:57:53.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:53.389 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":82,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:07.237328+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 msgr2=0x7f62100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 0x7f62100a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6218000c00 tx=0x7f6218004f10 comp rx=0 tx=0).stop 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 msgr2=0x7f622819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f622000cad0 tx=0x7f622000ced0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 shutdown_connections 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f62100a2800 0x7f62100a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62281022b0 0x7f622819f080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62281018f0 0x7f6228198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 --2- 172.21.15.150:0/3198323282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6228100700 0x7f6228197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 >> 172.21.15.150:0/3198323282 conn(0x7f62280fbc70 msgr2=0x7f6228104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 shutdown_connections 2024-03-18T15:57:53.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.390+0000 7f622e7cd640 1 -- 172.21.15.150:0/3198323282 wait complete. 2024-03-18T15:57:53.403 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:57:53.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:57:53.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 -- 172.21.15.150:0/4217911896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c41018d0 msgr2=0x7f59c4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/4217911896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c41018d0 0x7f59c4101d50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009900 tx=0x7f59b802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 -- 172.21.15.150:0/4217911896 shutdown_connections 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/4217911896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c4102290 0x7f59c4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/4217911896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c41018d0 0x7f59c4101d50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/4217911896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 0x7f59c4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 -- 172.21.15.150:0/4217911896 >> 172.21.15.150:0/4217911896 conn(0x7f59c40fbc70 msgr2=0x7f59c40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 -- 172.21.15.150:0/4217911896 shutdown_connections 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.510+0000 7f59c8a0b640 1 -- 172.21.15.150:0/4217911896 wait complete. 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 Processor -- start 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 -- start start 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 0x7f59c4197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 0x7f59c4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59c410c340 con 0x7f59c4102290 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f59c410c1c0 con 0x7f59c41018d0 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c8a0b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f59c410c4c0 con 0x7f59c41006e0 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 0x7f59c4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60626/0 (socket says 172.21.15.150:60626) 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c1d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 0x7f59c4198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:47216/0 (socket says 172.21.15.150:47216) 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c2d9d640 1 -- 172.21.15.150:0/2314253635 learned_addr learned my addr 172.21.15.150:0/2314253635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.511+0000 7f59c259c640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 0x7f59c4197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 -- 172.21.15.150:0/2314253635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 msgr2=0x7f59c4197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 0x7f59c4197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 -- 172.21.15.150:0/2314253635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 msgr2=0x7f59c4198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 0x7f59c4198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 -- 172.21.15.150:0/2314253635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c419f7a0 con 0x7f59c4102290 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c2d9d640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009fd0 tx=0x7f59bc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc017070 con 0x7f59c4102290 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59bc003e50 con 0x7f59c4102290 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c419fa90 con 0x7f59c4102290 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc0106c0 con 0x7f59c4102290 2024-03-18T15:57:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.512+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c419ff60 con 0x7f59c4102290 2024-03-18T15:57:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.513+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c41a01f0 con 0x7f59c4102290 2024-03-18T15:57:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.515+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f59bc003970 con 0x7f59c4102290 2024-03-18T15:57:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.515+0000 7f59b37fe640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 0x7f59ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.515+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f59bc0ce2a0 con 0x7f59c4102290 2024-03-18T15:57:53.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.515+0000 7f59c259c640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 0x7f59ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.516+0000 7f59c259c640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 0x7f59ac0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f59c4101730 tx=0x7f59b4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.519+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59bc09b190 con 0x7f59c4102290 2024-03-18T15:57:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.701+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59c41a0460 con 0x7f59c4102290 2024-03-18T15:57:53.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.702+0000 7f59b37fe640 1 -- 172.21.15.150:0/2314253635 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f59bc09f230 con 0x7f59c4102290 2024-03-18T15:57:53.703 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:57:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 msgr2=0x7f59ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 0x7f59ac0a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f59c4101730 tx=0x7f59b4005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 msgr2=0x7f59c419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009fd0 tx=0x7f59bc00b260 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 shutdown_connections 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f59ac0a2800 0x7f59ac0a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c4102290 0x7f59c419f130 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59c41018d0 0x7f59c4198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 --2- 172.21.15.150:0/2314253635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f59c41006e0 0x7f59c4197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 >> 172.21.15.150:0/2314253635 conn(0x7f59c40fbc70 msgr2=0x7f59c40fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 shutdown_connections 2024-03-18T15:57:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.704+0000 7f59c8a0b640 1 -- 172.21.15.150:0/2314253635 wait complete. 2024-03-18T15:57:53.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T15:57:53.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 -- 172.21.15.150:0/657283485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc1033c0 msgr2=0x7fcdfc109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 --2- 172.21.15.150:0/657283485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc109c50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009ca0 tx=0x7fcdf402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 -- 172.21.15.150:0/657283485 shutdown_connections 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 --2- 172.21.15.150:0/657283485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc109c50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 --2- 172.21.15.150:0/657283485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc102a00 0x7fcdfc102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 --2- 172.21.15.150:0/657283485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 -- 172.21.15.150:0/657283485 >> 172.21.15.150:0/657283485 conn(0x7fcdfc0fcd80 msgr2=0x7fcdfc0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 -- 172.21.15.150:0/657283485 shutdown_connections 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.830+0000 7fce043cd640 1 -- 172.21.15.150:0/657283485 wait complete. 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 Processor -- start 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 -- start start 2024-03-18T15:57:53.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc10d490 con 0x7fcdfc102a00 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc10d310 con 0x7fcdfc101810 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce043cd640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc10d610 con 0x7fcdfc1033c0 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60650/0 (socket says 172.21.15.150:60650) 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 -- 172.21.15.150:0/4130185211 learned_addr learned my addr 172.21.15.150:0/4130185211 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce0296a640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce02169640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 -- 172.21.15.150:0/4130185211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc1033c0 msgr2=0x7fcdfc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 -- 172.21.15.150:0/4130185211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 msgr2=0x7fcdfc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 -- 172.21.15.150:0/4130185211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdfc19f6f0 con 0x7fcdfc102a00 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce02169640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.831+0000 7fce01968640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e9f0 tx=0x7fcdec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.832+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00cd80 con 0x7fcdfc102a00 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.832+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdec005500 con 0x7fcdfc102a00 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.832+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc19f9e0 con 0x7fcdfc102a00 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.832+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec010690 con 0x7fcdfc102a00 2024-03-18T15:57:53.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.832+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc19ff30 con 0x7fcdfc102a00 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.833+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdfc108b30 con 0x7fcdfc102a00 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.835+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fcdec0046c0 con 0x7fcdfc102a00 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.836+0000 7fcdf37fe640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 0x7fcde80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.836+0000 7fce02169640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 0x7fcde80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.836+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcdec0cebe0 con 0x7fcdfc102a00 2024-03-18T15:57:53.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.836+0000 7fce02169640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 0x7fcde80a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc102860 tx=0x7fcdf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:53.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:53.839+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdec053740 con 0x7fcdfc102a00 2024-03-18T15:57:54.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.027+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcdfc02d1b0 con 0x7fcdfc102a00 2024-03-18T15:57:54.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.027+0000 7fcdf37fe640 1 -- 172.21.15.150:0/4130185211 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fcdec09c120 con 0x7fcdfc102a00 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T15:57:54.029 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 msgr2=0x7fcde80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 0x7fcde80a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc102860 tx=0x7fcdf8005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 msgr2=0x7fcdfc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e9f0 tx=0x7fcdec00edf0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 shutdown_connections 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fcde80a2880 0x7fcde80a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc1033c0 0x7fcdfc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdfc102a00 0x7fcdfc198010 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 --2- 172.21.15.150:0/4130185211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdfc101810 0x7fcdfc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 >> 172.21.15.150:0/4130185211 conn(0x7fcdfc0fcd80 msgr2=0x7fcdfc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 shutdown_connections 2024-03-18T15:57:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.030+0000 7fce043cd640 1 -- 172.21.15.150:0/4130185211 wait complete. 2024-03-18T15:57:54.043 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T15:57:54.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- 172.21.15.150:0/1247406100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76541074e0 msgr2=0x7f7654109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 --2- 172.21.15.150:0/1247406100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76541074e0 0x7f7654109930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f764c009ca0 tx=0x7f764c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- 172.21.15.150:0/1247406100 shutdown_connections 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 --2- 172.21.15.150:0/1247406100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76541074e0 0x7f7654109930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 --2- 172.21.15.150:0/1247406100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7654104bb0 0x7f7654106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 --2- 172.21.15.150:0/1247406100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654102280 0x7f7654104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- 172.21.15.150:0/1247406100 >> 172.21.15.150:0/1247406100 conn(0x7f76540fbc90 msgr2=0x7f76540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- 172.21.15.150:0/1247406100 shutdown_connections 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- 172.21.15.150:0/1247406100 wait complete. 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 Processor -- start 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.152+0000 7f765abc4640 1 -- start start 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 0x7f7654197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76541074e0 0x7f765419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f765410d480 con 0x7f7654102280 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f765410d300 con 0x7f7654104bb0 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f765410d600 con 0x7f76541074e0 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:54.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60656/0 (socket says 172.21.15.150:60656) 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 -- 172.21.15.150:0/3312486265 learned_addr learned my addr 172.21.15.150:0/3312486265 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7653fff640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 0x7f7654197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7659161640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76541074e0 0x7f765419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 -- 172.21.15.150:0/3312486265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76541074e0 msgr2=0x7f765419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76541074e0 0x7f765419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 -- 172.21.15.150:0/3312486265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 msgr2=0x7f7654197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 0x7f7654197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 -- 172.21.15.150:0/3312486265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765419f6c0 con 0x7f7654102280 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7653fff640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 0x7f7654197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7658960640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f764800e9f0 tx=0x7f764800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764800cd80 con 0x7f7654102280 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7648010910 con 0x7f7654102280 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765419f9b0 con 0x7f7654102280 2024-03-18T15:57:54.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.153+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648018990 con 0x7f7654102280 2024-03-18T15:57:54.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.154+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765419ff00 con 0x7f7654102280 2024-03-18T15:57:54.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.155+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7654067d40 con 0x7f7654102280 2024-03-18T15:57:54.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.156+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7648010ac0 con 0x7f7654102280 2024-03-18T15:57:54.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.156+0000 7f7651ffb640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:54.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.156+0000 7f7653fff640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:54.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.156+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f76480ce910 con 0x7f7654102280 2024-03-18T15:57:54.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.157+0000 7f7653fff640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 0x7f76440a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7654198fb0 tx=0x7f7640005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:54.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.161+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f764809b800 con 0x7f7654102280 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1940686425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3198323282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2314253635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:54.343 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4130185211' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:54.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:54.342+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f76540fd780 con 0x7f7654102280 2024-03-18T15:57:54.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1940686425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3198323282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2314253635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:54.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4130185211' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1940686425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:54.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3198323282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:54.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:54.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2314253635' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:57:54.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4130185211' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:55.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.148+0000 7f7651ffb640 1 -- 172.21.15.150:0/3312486265 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f764809f8a0 con 0x7f7654102280 2024-03-18T15:57:55.149 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 msgr2=0x7f76440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 0x7f76440a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7654198fb0 tx=0x7f7640005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 msgr2=0x7f7654197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f764800e9f0 tx=0x7f764800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 shutdown_connections 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f76541074e0 0x7f765419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7654104bb0 0x7f7654197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 --2- 172.21.15.150:0/3312486265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7654102280 0x7f7654197aa0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.150+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 >> 172.21.15.150:0/3312486265 conn(0x7f76540fbc90 msgr2=0x7f7654102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.151+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 shutdown_connections 2024-03-18T15:57:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.151+0000 7f765abc4640 1 -- 172.21.15.150:0/3312486265 wait complete. 2024-03-18T15:57:55.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T15:57:55.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- 172.21.15.150:0/3908443270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 msgr2=0x7f46c0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- 172.21.15.150:0/3908443270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0100b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009900 tx=0x7f46b002bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- 172.21.15.150:0/3908443270 shutdown_connections 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- 172.21.15.150:0/3908443270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c0102290 0x7f46c0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- 172.21.15.150:0/3908443270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c01018d0 0x7f46c0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- 172.21.15.150:0/3908443270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0100b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- 172.21.15.150:0/3908443270 >> 172.21.15.150:0/3908443270 conn(0x7f46c00fbc70 msgr2=0x7f46c00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- 172.21.15.150:0/3908443270 shutdown_connections 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- 172.21.15.150:0/3908443270 wait complete. 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 Processor -- start 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- start start 2024-03-18T15:57:55.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 0x7f46c01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c0102290 0x7f46c019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010c350 con 0x7f46c01006e0 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010c1d0 con 0x7f46c01018d0 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46c55f3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010c4d0 con 0x7f46c0102290 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.274+0000 7f46beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60672/0 (socket says 172.21.15.150:60672) 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 -- 172.21.15.150:0/4241880242 learned_addr learned my addr 172.21.15.150:0/4241880242 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46b7fff640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 0x7f46c01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46bf7fe640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c0102290 0x7f46c019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 -- 172.21.15.150:0/4241880242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c0102290 msgr2=0x7f46c019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c0102290 0x7f46c019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 -- 172.21.15.150:0/4241880242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 msgr2=0x7f46c01980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 0x7f46c01980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 -- 172.21.15.150:0/4241880242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c019f790 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46b7fff640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 0x7f46c01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46beffd640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0041070 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46b0004450 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c019fa20 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0034440 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.275+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c019ff10 con 0x7f46c01006e0 2024-03-18T15:57:55.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.276+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c01a00e0 con 0x7f46c01006e0 2024-03-18T15:57:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.277+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f46b0044070 con 0x7f46c01006e0 2024-03-18T15:57:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.278+0000 7f46bcff9640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 0x7f46ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.278+0000 7f46b7fff640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 0x7f46ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.278+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f46b00f0420 con 0x7f46c01006e0 2024-03-18T15:57:55.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.278+0000 7f46b7fff640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 0x7f46ac0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f46c0199080 tx=0x7f46a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:55.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.282+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f46b00bd210 con 0x7f46c01006e0 2024-03-18T15:57:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:57:55.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:55.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:55.463+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f46c01a02b0 con 0x7f46c01006e0 2024-03-18T15:57:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:57:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:55.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:55.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:57:55.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:56.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.154+0000 7f46bcff9640 1 -- 172.21.15.150:0/4241880242 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f46b00c12b0 con 0x7f46c01006e0 2024-03-18T15:57:56.155 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-03-18T15:57:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 msgr2=0x7f46ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 0x7f46ac0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f46c0199080 tx=0x7f46a8005f90 comp rx=0 tx=0).stop 2024-03-18T15:57:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 msgr2=0x7f46c0197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f46b0000c00 tx=0x7f46b002c640 comp rx=0 tx=0).stop 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 shutdown_connections 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f46ac0a2800 0x7f46ac0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46c0102290 0x7f46c019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c01018d0 0x7f46c01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 --2- 172.21.15.150:0/4241880242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c01006e0 0x7f46c0197b70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 >> 172.21.15.150:0/4241880242 conn(0x7f46c00fbc70 msgr2=0x7f46c00fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 shutdown_connections 2024-03-18T15:57:56.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.157+0000 7f46c55f3640 1 -- 172.21.15.150:0/4241880242 wait complete. 2024-03-18T15:57:56.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T15:57:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 -- 172.21.15.150:0/337231645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 msgr2=0x7ff9a8102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/337231645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8102e80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff998009900 tx=0x7ff99802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 -- 172.21.15.150:0/337231645 shutdown_connections 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/337231645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a81033c0 0x7ff9a8109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/337231645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8102e80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/337231645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a8101810 0x7ff9a8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 -- 172.21.15.150:0/337231645 >> 172.21.15.150:0/337231645 conn(0x7ff9a80fcd80 msgr2=0x7ff9a80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 -- 172.21.15.150:0/337231645 shutdown_connections 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.285+0000 7ff9b00f8640 1 -- 172.21.15.150:0/337231645 wait complete. 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 Processor -- start 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 -- start start 2024-03-18T15:57:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 0x7ff9a8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a81033c0 0x7ff9a819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810d500 con 0x7ff9a8102a00 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810d380 con 0x7ff9a81033c0 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a810d680 con 0x7ff9a8101810 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60700/0 (socket says 172.21.15.150:60700) 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 -- 172.21.15.150:0/452636263 learned_addr learned my addr 172.21.15.150:0/452636263 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ade94640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 0x7ff9a8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 -- 172.21.15.150:0/452636263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 msgr2=0x7ff9a8197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 0x7ff9a8197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 -- 172.21.15.150:0/452636263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a81033c0 msgr2=0x7ff9a819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a81033c0 0x7ff9a819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:56.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 -- 172.21.15.150:0/452636263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a819f770 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ade94640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 0x7ff9a8197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9ad693640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff998009230 tx=0x7ff998004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998041070 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9980054e0 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980346a0 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.286+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a819fa00 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.287+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a819ff40 con 0x7ff9a8102a00 2024-03-18T15:57:56.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.288+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a8108ad0 con 0x7ff9a8102a00 2024-03-18T15:57:56.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.289+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff998044070 con 0x7ff9a8102a00 2024-03-18T15:57:56.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.290+0000 7ff99effd640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 0x7ff9940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:56.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.290+0000 7ff9ade94640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 0x7ff9940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:56.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.290+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7ff9980f0a70 con 0x7ff9a8102a00 2024-03-18T15:57:56.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.290+0000 7ff9ade94640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 0x7ff9940a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006fd0 tx=0x7ff9a4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:56.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.294+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9980bdcc0 con 0x7ff9a8102a00 2024-03-18T15:57:56.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:56 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:57:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:56 smithi191 ceph-mon[36784]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:57:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:56 smithi191 ceph-mon[36784]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:56.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:56 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:57:56.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:56.510+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff9a8062450 con 0x7ff9a8102a00 2024-03-18T15:57:56.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:56 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:57:56.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:56 smithi156 ceph-mon[35919]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:57:56.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:56 smithi156 ceph-mon[35919]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:56.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:56.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:56 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:57:56.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:56 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3312486265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T15:57:56.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:56 smithi150 ceph-mon[31946]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T15:57:56.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:56 smithi150 ceph-mon[31946]: pgmap v283: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:56.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:57:56.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:56 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T15:57:57.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.156+0000 7ff99effd640 1 -- 172.21.15.150:0/452636263 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff9980bd5c0 con 0x7ff9a8102a00 2024-03-18T15:57:57.202 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 msgr2=0x7ff9940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 0x7ff9940a4d40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006fd0 tx=0x7ff9a4008040 comp rx=0 tx=0).stop 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 msgr2=0x7ff9a8198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff998009230 tx=0x7ff998004040 comp rx=0 tx=0).stop 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 shutdown_connections 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff9940a2880 0x7ff9940a4d40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9a81033c0 0x7ff9a819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8102a00 0x7ff9a8198090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 --2- 172.21.15.150:0/452636263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9a8101810 0x7ff9a8197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.159+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 >> 172.21.15.150:0/452636263 conn(0x7ff9a80fcd80 msgr2=0x7ff9a8105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.160+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 shutdown_connections 2024-03-18T15:57:57.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.160+0000 7ff9b00f8640 1 -- 172.21.15.150:0/452636263 wait complete. 2024-03-18T15:57:57.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:57:57.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 -- 172.21.15.150:0/1615602000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c41018f0 msgr2=0x7f29c4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 --2- 172.21.15.150:0/1615602000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c41018f0 0x7f29c4101d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009900 tx=0x7f29b802bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 -- 172.21.15.150:0/1615602000 shutdown_connections 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 --2- 172.21.15.150:0/1615602000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29c41022b0 0x7f29c4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 --2- 172.21.15.150:0/1615602000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c41018f0 0x7f29c4101d70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 --2- 172.21.15.150:0/1615602000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c4100700 0x7f29c4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 -- 172.21.15.150:0/1615602000 >> 172.21.15.150:0/1615602000 conn(0x7f29c40fbc70 msgr2=0x7f29c40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 -- 172.21.15.150:0/1615602000 shutdown_connections 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.307+0000 7f29ca080640 1 -- 172.21.15.150:0/1615602000 wait complete. 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 Processor -- start 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 -- start start 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c41018f0 0x7f29c419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29c41022b0 0x7f29c41a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29c410c380 con 0x7f29c4100700 2024-03-18T15:57:57.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f29c410c200 con 0x7f29c41018f0 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29ca080640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f29c410c500 con 0x7f29c41022b0 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60714/0 (socket says 172.21.15.150:60714) 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 -- 172.21.15.150:0/3038389240 learned_addr learned my addr 172.21.15.150:0/3038389240 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c2ffd640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c41018f0 0x7f29c419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 -- 172.21.15.150:0/3038389240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29c41022b0 msgr2=0x7f29c41a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29c41022b0 0x7f29c41a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 -- 172.21.15.150:0/3038389240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c41018f0 msgr2=0x7f29c419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c41018f0 0x7f29c419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.308+0000 7f29c37fe640 1 -- 172.21.15.150:0/3038389240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c41a18c0 con 0x7f29c4100700 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29c37fe640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c970 tx=0x7f29b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b4007c10 con 0x7f29c4100700 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29b4007db0 con 0x7f29c4100700 2024-03-18T15:57:57.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c41a1b50 con 0x7f29c4100700 2024-03-18T15:57:57.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b400f940 con 0x7f29c4100700 2024-03-18T15:57:57.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.309+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c41a2070 con 0x7f29c4100700 2024-03-18T15:57:57.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.310+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c41079c0 con 0x7f29c4100700 2024-03-18T15:57:57.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.311+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f29b400fae0 con 0x7f29c4100700 2024-03-18T15:57:57.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.311+0000 7f29c0ff9640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 0x7f29ac0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.312+0000 7f29c2ffd640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 0x7f29ac0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.312+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f29b40cef20 con 0x7f29c4100700 2024-03-18T15:57:57.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.312+0000 7f29c2ffd640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 0x7f29ac0a4cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f29b802c7b0 tx=0x7f29b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.315+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29b409c230 con 0x7f29c4100700 2024-03-18T15:57:57.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:57 smithi150 ceph-mon[31946]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:57:57.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:57 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:57:57.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:57 smithi150 ceph-mon[31946]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:57:57.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:57 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:57:57.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:57 smithi150 ceph-mon[31946]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-18T15:57:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:57 smithi191 ceph-mon[36784]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:57:57.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:57 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:57:57.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:57 smithi191 ceph-mon[36784]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:57:57.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:57 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:57:57.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:57 smithi191 ceph-mon[36784]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-18T15:57:57.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.494+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29c402d350 con 0x7f29c4100700 2024-03-18T15:57:57.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.495+0000 7f29c0ff9640 1 -- 172.21.15.150:0/3038389240 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22288 (secure 0 0 0) 0x7f29b409bb30 con 0x7f29c4100700 2024-03-18T15:57:57.497 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:57.497 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":85,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:56.511654+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 msgr2=0x7f29ac0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 0x7f29ac0a4cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f29b802c7b0 tx=0x7f29b8005fd0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 msgr2=0x7f29c406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c970 tx=0x7f29b400ce30 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 shutdown_connections 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ac0a2830 0x7f29ac0a4cf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29c41022b0 0x7f29c41a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c41018f0 0x7f29c419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 --2- 172.21.15.150:0/3038389240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c4100700 0x7f29c406a9c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.497+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 >> 172.21.15.150:0/3038389240 conn(0x7f29c40fbc70 msgr2=0x7f29c4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.498+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 shutdown_connections 2024-03-18T15:57:57.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.498+0000 7f29ca080640 1 -- 172.21.15.150:0/3038389240 wait complete. 2024-03-18T15:57:57.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:57:57.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:57 smithi156 ceph-mon[35919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T15:57:57.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:57 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4241880242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T15:57:57.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:57 smithi156 ceph-mon[35919]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T15:57:57.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:57 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T15:57:57.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:57 smithi156 ceph-mon[35919]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.610+0000 7fde98d53640 1 -- 172.21.15.150:0/3991575781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941033b0 msgr2=0x7fde94109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.610+0000 7fde98d53640 1 --2- 172.21.15.150:0/3991575781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941033b0 0x7fde94109c40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009ca0 tx=0x7fde8c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 -- 172.21.15.150:0/3991575781 shutdown_connections 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 --2- 172.21.15.150:0/3991575781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941033b0 0x7fde94109c40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 --2- 172.21.15.150:0/3991575781 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde941029f0 0x7fde94102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 --2- 172.21.15.150:0/3991575781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde94101800 0x7fde94101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 -- 172.21.15.150:0/3991575781 >> 172.21.15.150:0/3991575781 conn(0x7fde940fcd90 msgr2=0x7fde940ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 -- 172.21.15.150:0/3991575781 shutdown_connections 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 -- 172.21.15.150:0/3991575781 wait complete. 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 Processor -- start 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 -- start start 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.611+0000 7fde98d53640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde94101800 0x7fde94197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde941033b0 0x7fde9419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde9410d460 con 0x7fde941029f0 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde9410d2e0 con 0x7fde941033b0 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fde9410d5e0 con 0x7fde94101800 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60730/0 (socket says 172.21.15.150:60730) 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 -- 172.21.15.150:0/1625082151 learned_addr learned my addr 172.21.15.150:0/1625082151 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde9259c640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde94101800 0x7fde94197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde92d9d640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde941033b0 0x7fde9419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 -- 172.21.15.150:0/1625082151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde94101800 msgr2=0x7fde94197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde94101800 0x7fde94197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 -- 172.21.15.150:0/1625082151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde941033b0 msgr2=0x7fde9419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde941033b0 0x7fde9419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 -- 172.21.15.150:0/1625082151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9419f6c0 con 0x7fde941029f0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde91d9b640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d950 tx=0x7fde8800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88017070 con 0x7fde941029f0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9419f9b0 con 0x7fde941029f0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde88009c30 con 0x7fde941029f0 2024-03-18T15:57:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8800f6b0 con 0x7fde941029f0 2024-03-18T15:57:57.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.612+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9419ff00 con 0x7fde941029f0 2024-03-18T15:57:57.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.614+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde94108b20 con 0x7fde941029f0 2024-03-18T15:57:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.615+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fde880109d0 con 0x7fde941029f0 2024-03-18T15:57:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.616+0000 7fde837fe640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 0x7fde7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.616+0000 7fde9259c640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 0x7fde7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.616+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fde880ce580 con 0x7fde941029f0 2024-03-18T15:57:57.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.616+0000 7fde9259c640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 0x7fde7c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fde94102850 tx=0x7fde8400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.620+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde88009de0 con 0x7fde941029f0 2024-03-18T15:57:57.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.798+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde9402d330 con 0x7fde941029f0 2024-03-18T15:57:57.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.800+0000 7fde837fe640 1 -- 172.21.15.150:0/1625082151 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22288 (secure 0 0 0) 0x7fde8809b770 con 0x7fde941029f0 2024-03-18T15:57:57.801 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:57:57.802 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":85,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:56.511654+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:57:57.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 msgr2=0x7fde7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 0x7fde7c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fde94102850 tx=0x7fde8400a2d0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 msgr2=0x7fde94197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde8800d950 tx=0x7fde8800de10 comp rx=0 tx=0).stop 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 shutdown_connections 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fde7c0a2800 0x7fde7c0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde941033b0 0x7fde9419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde941029f0 0x7fde94197fe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 --2- 172.21.15.150:0/1625082151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde94101800 0x7fde94197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 >> 172.21.15.150:0/1625082151 conn(0x7fde940fcd90 msgr2=0x7fde94105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 shutdown_connections 2024-03-18T15:57:57.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.802+0000 7fde98d53640 1 -- 172.21.15.150:0/1625082151 wait complete. 2024-03-18T15:57:57.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 -- 172.21.15.150:0/2989137864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c100a20 msgr2=0x7f604c100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 --2- 172.21.15.150:0/2989137864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c100a20 0x7f604c100e80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f603c009900 tx=0x7f603c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 -- 172.21.15.150:0/2989137864 shutdown_connections 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 --2- 172.21.15.150:0/2989137864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c1013c0 0x7f604c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 --2- 172.21.15.150:0/2989137864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c100a20 0x7f604c100e80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 --2- 172.21.15.150:0/2989137864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 -- 172.21.15.150:0/2989137864 >> 172.21.15.150:0/2989137864 conn(0x7f604c0fbc80 msgr2=0x7f604c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 -- 172.21.15.150:0/2989137864 shutdown_connections 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.918+0000 7f6053997640 1 -- 172.21.15.150:0/2989137864 wait complete. 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 Processor -- start 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 -- start start 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 0x7f604c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f604c10f6b0 con 0x7f604c1013c0 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f604c10f530 con 0x7f604c1000c0 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6053997640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f604c10f830 con 0x7f604c100a20 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60752/0 (socket says 172.21.15.150:60752) 2024-03-18T15:57:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 -- 172.21.15.150:0/964414899 learned_addr learned my addr 172.21.15.150:0/964414899 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051733640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6050f32640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 0x7f604c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 -- 172.21.15.150:0/964414899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 msgr2=0x7f604c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 0x7f604c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 -- 172.21.15.150:0/964414899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 msgr2=0x7f604c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 -- 172.21.15.150:0/964414899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604c1a19d0 con 0x7f604c1013c0 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051733640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6050f32640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 0x7f604c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T15:57:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.919+0000 7f6051f34640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f604400b970 tx=0x7f604400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.920+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044019070 con 0x7f604c1013c0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.920+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f604400cb40 con 0x7f604c1013c0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.920+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604c1a1cc0 con 0x7f604c1013c0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.920+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044007650 con 0x7f604c1013c0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.920+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604c1a21c0 con 0x7f604c1013c0 2024-03-18T15:57:57.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.921+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604c10b050 con 0x7f604c1013c0 2024-03-18T15:57:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.922+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f604401c070 con 0x7f604c1013c0 2024-03-18T15:57:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.922+0000 7f60427fc640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 0x7f60380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.922+0000 7f6051733640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 0x7f60380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.922+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f60440ce3f0 con 0x7f604c1013c0 2024-03-18T15:57:57.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.923+0000 7f6051733640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 0x7f60380a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f60480058c0 tx=0x7f6048005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:57.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:57.926+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f604409b5e0 con 0x7f604c1013c0 2024-03-18T15:57:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.140+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f604c02d3d0 con 0x7f604c1013c0 2024-03-18T15:57:58.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.170+0000 7f60427fc640 1 -- 172.21.15.150:0/964414899 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f604409aee0 con 0x7f604c1013c0 2024-03-18T15:57:58.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 msgr2=0x7f60380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 0x7f60380a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f60480058c0 tx=0x7f6048005d30 comp rx=0 tx=0).stop 2024-03-18T15:57:58.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 msgr2=0x7f604c1a1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f604400b970 tx=0x7f604400be30 comp rx=0 tx=0).stop 2024-03-18T15:57:58.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 shutdown_connections 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60380a2800 0x7f60380a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f604c1013c0 0x7f604c1a1360 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f604c100a20 0x7f604c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 --2- 172.21.15.150:0/964414899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604c1000c0 0x7f604c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 >> 172.21.15.150:0/964414899 conn(0x7f604c0fbc80 msgr2=0x7f604c0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 shutdown_connections 2024-03-18T15:57:58.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.173+0000 7f6053997640 1 -- 172.21.15.150:0/964414899 wait complete. 2024-03-18T15:57:58.186 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-18T15:57:58.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-18T15:57:58.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 -- 172.21.15.150:0/1664939205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1006e0 msgr2=0x7f79ec100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 --2- 172.21.15.150:0/1664939205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1006e0 0x7f79ec100b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009900 tx=0x7f79dc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:57:58.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 -- 172.21.15.150:0/1664939205 shutdown_connections 2024-03-18T15:57:58.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 --2- 172.21.15.150:0/1664939205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec102290 0x7f79ec108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 --2- 172.21.15.150:0/1664939205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec1018d0 0x7f79ec101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 --2- 172.21.15.150:0/1664939205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1006e0 0x7f79ec100b00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 -- 172.21.15.150:0/1664939205 >> 172.21.15.150:0/1664939205 conn(0x7f79ec0fbc70 msgr2=0x7f79ec0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 -- 172.21.15.150:0/1664939205 shutdown_connections 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.289+0000 7f79f2634640 1 -- 172.21.15.150:0/1664939205 wait complete. 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 Processor -- start 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 -- start start 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec1006e0 0x7f79ec0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec102290 0x7f79ec19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec10c360 con 0x7f79ec1018d0 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec10c1e0 con 0x7f79ec102290 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f2634640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec10c4e0 con 0x7f79ec1006e0 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79ebfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec1006e0 0x7f79ec0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60764/0 (socket says 172.21.15.150:60764) 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 -- 172.21.15.150:0/204199161 learned_addr learned my addr 172.21.15.150:0/204199161 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79f0bd1640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec102290 0x7f79ec19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 -- 172.21.15.150:0/204199161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec1006e0 msgr2=0x7f79ec0694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec1006e0 0x7f79ec0694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 -- 172.21.15.150:0/204199161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec102290 msgr2=0x7f79ec19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec102290 0x7f79ec19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.290+0000 7f79eb7fe640 1 -- 172.21.15.150:0/204199161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ec19f770 con 0x7f79ec1018d0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79eb7fe640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d950 tx=0x7f79e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0017070 con 0x7f79ec1018d0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0009c30 con 0x7f79ec1018d0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ec19fa00 con 0x7f79ec1018d0 2024-03-18T15:57:58.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e000f6b0 con 0x7f79ec1018d0 2024-03-18T15:57:58.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.291+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ec19ff50 con 0x7f79ec1018d0 2024-03-18T15:57:58.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.292+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ec107ad0 con 0x7f79ec1018d0 2024-03-18T15:57:58.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.293+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f79e00109d0 con 0x7f79ec1018d0 2024-03-18T15:57:58.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.293+0000 7f79e97fa640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 0x7f79d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.294+0000 7f79ebfff640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 0x7f79d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.294+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f79e00ce530 con 0x7f79ec1018d0 2024-03-18T15:57:58.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.294+0000 7f79ebfff640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 0x7f79d40a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000c00 tx=0x7f79dc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:58.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.298+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79e0009de0 con 0x7f79ec1018d0 2024-03-18T15:57:58.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:57:58.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:57:58.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-18T15:57:58.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-18T15:57:58.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: pgmap v286: 97 pgs: 25 creating+peering, 18 active+clean, 54 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:58.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3038389240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1625082151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:57:58.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:58.512+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f79ec02d400 con 0x7f79ec1018d0 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: pgmap v286: 97 pgs: 25 creating+peering, 18 active+clean, 54 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3038389240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1625082151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/452636263' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: pgmap v286: 97 pgs: 25 creating+peering, 18 active+clean, 54 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3038389240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1625082151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:57:58.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T15:57:59.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.171+0000 7f79e97fa640 1 -- 172.21.15.150:0/204199161 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f79e009b720 con 0x7f79ec1018d0 2024-03-18T15:57:59.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 msgr2=0x7f79d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 0x7f79d40a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000c00 tx=0x7f79dc005fd0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 msgr2=0x7f79ec19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d950 tx=0x7f79e000de10 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 shutdown_connections 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79d40a2800 0x7f79d40a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79ec102290 0x7f79ec19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79ec1018d0 0x7f79ec19c840 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 --2- 172.21.15.150:0/204199161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79ec1006e0 0x7f79ec0694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.174+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 >> 172.21.15.150:0/204199161 conn(0x7f79ec0fbc70 msgr2=0x7f79ec104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.175+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 shutdown_connections 2024-03-18T15:57:59.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.175+0000 7f79f2634640 1 -- 172.21.15.150:0/204199161 wait complete. 2024-03-18T15:57:59.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 -- 172.21.15.150:0/2314107979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 msgr2=0x7f5bd8109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 --2- 172.21.15.150:0/2314107979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd8109db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009cd0 tx=0x7f5bd002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 -- 172.21.15.150:0/2314107979 shutdown_connections 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 --2- 172.21.15.150:0/2314107979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd8109db0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 --2- 172.21.15.150:0/2314107979 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd8102b60 0x7f5bd8102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 --2- 172.21.15.150:0/2314107979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd80ffaf0 0x7f5bd80fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 -- 172.21.15.150:0/2314107979 >> 172.21.15.150:0/2314107979 conn(0x7f5bd80fb3e0 msgr2=0x7f5bd80fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 -- 172.21.15.150:0/2314107979 shutdown_connections 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.304+0000 7f5bdf510640 1 -- 172.21.15.150:0/2314107979 wait complete. 2024-03-18T15:57:59.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 Processor -- start 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 -- start start 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd80ffaf0 0x7f5bd8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8102b60 0x7f5bd81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd810d510 con 0x7f5bd8103520 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd810d390 con 0x7f5bd8102b60 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdf510640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd810d690 con 0x7f5bd80ffaf0 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60774/0 (socket says 172.21.15.150:60774) 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 -- 172.21.15.150:0/1192243283 learned_addr learned my addr 172.21.15.150:0/1192243283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdd2ac640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd80ffaf0 0x7f5bd8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 -- 172.21.15.150:0/1192243283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd80ffaf0 msgr2=0x7f5bd8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd80ffaf0 0x7f5bd8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 -- 172.21.15.150:0/1192243283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8102b60 msgr2=0x7f5bd81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bdcaab640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8102b60 0x7f5bd81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8102b60 0x7f5bd81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 -- 172.21.15.150:0/1192243283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd819f780 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.305+0000 7f5bddaad640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0005e30 tx=0x7f5bd0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.306+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0039040 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.306+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd0044070 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.306+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd819fa10 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.306+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00042a0 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.306+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd819ff00 con 0x7f5bd8103520 2024-03-18T15:57:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.307+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd8108c90 con 0x7f5bd8103520 2024-03-18T15:57:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.308+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f5bd0032690 con 0x7f5bd8103520 2024-03-18T15:57:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.308+0000 7f5bce7fc640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 0x7f5bc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.308+0000 7f5bdd2ac640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 0x7f5bc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.308+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5bd00f0d90 con 0x7f5bd8103520 2024-03-18T15:57:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.309+0000 7f5bdd2ac640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 0x7f5bc40a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8069b30 tx=0x7f5bd4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.313+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bd00bdf80 con 0x7f5bd8103520 2024-03-18T15:57:59.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:57:59.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: Cluster is now healthy 2024-03-18T15:57:59.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:57:59.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:active 2024-03-18T15:57:59.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:57:59.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:57:59.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:57:59 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T15:57:59.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:57:59.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: Cluster is now healthy 2024-03-18T15:57:59.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:57:59.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:active 2024-03-18T15:57:59.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:57:59.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:57:59.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:57:59 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T15:57:59.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:57:59.526+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f5bd802d1b0 con 0x7f5bd8103520 2024-03-18T15:57:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T15:57:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: Cluster is now healthy 2024-03-18T15:57:59.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/964414899' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T15:57:59.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] up:active 2024-03-18T15:57:59.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:57:59.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T15:57:59.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:57:59 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-18T15:58:00.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.183+0000 7f5bce7fc640 1 -- 172.21.15.150:0/1192243283 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f5bd00bd880 con 0x7f5bd8103520 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 msgr2=0x7f5bc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 0x7f5bc40a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8069b30 tx=0x7f5bd4009110 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 msgr2=0x7f5bd819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0005e30 tx=0x7f5bd0032f40 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 shutdown_connections 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f5bc40a2800 0x7f5bc40a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd8103520 0x7f5bd819f110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd8102b60 0x7f5bd81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 --2- 172.21.15.150:0/1192243283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5bd80ffaf0 0x7f5bd8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.186+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 >> 172.21.15.150:0/1192243283 conn(0x7f5bd80fb3e0 msgr2=0x7f5bd8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.187+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 shutdown_connections 2024-03-18T15:58:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.187+0000 7f5bdf510640 1 -- 172.21.15.150:0/1192243283 wait complete. 2024-03-18T15:58:00.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.309+0000 7f8554220640 1 -- 172.21.15.150:0/782201063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c1006e0 msgr2=0x7f854c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.309+0000 7f8554220640 1 --2- 172.21.15.150:0/782201063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c1006e0 0x7f854c100b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8540009900 tx=0x7f854002bc30 comp rx=0 tx=0).stop 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- 172.21.15.150:0/782201063 shutdown_connections 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- 172.21.15.150:0/782201063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c102290 0x7f854c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- 172.21.15.150:0/782201063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1018d0 0x7f854c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- 172.21.15.150:0/782201063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c1006e0 0x7f854c100b00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- 172.21.15.150:0/782201063 >> 172.21.15.150:0/782201063 conn(0x7f854c0fbc70 msgr2=0x7f854c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- 172.21.15.150:0/782201063 shutdown_connections 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- 172.21.15.150:0/782201063 wait complete. 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 Processor -- start 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- start start 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1006e0 0x7f854c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c1018d0 0x7f854c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10d480 con 0x7f854c102290 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10d300 con 0x7f854c1006e0 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8554220640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10d600 con 0x7f854c1018d0 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f85527bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f85527bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60790/0 (socket says 172.21.15.150:60790) 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f8551fbc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1006e0 0x7f854c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.310+0000 7f85527bd640 1 -- 172.21.15.150:0/2097298708 learned_addr learned my addr 172.21.15.150:0/2097298708 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85517bb640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c1018d0 0x7f854c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 -- 172.21.15.150:0/2097298708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c1018d0 msgr2=0x7f854c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c1018d0 0x7f854c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 -- 172.21.15.150:0/2097298708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1006e0 msgr2=0x7f854c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1006e0 0x7f854c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 -- 172.21.15.150:0/2097298708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c19f760 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f85527bd640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f854400eb10 tx=0x7f854400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544004b20 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8544004cc0 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c19fa50 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544010760 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.311+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c19ffa0 con 0x7f854c102290 2024-03-18T15:58:00.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.312+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854c107a20 con 0x7f854c102290 2024-03-18T15:58:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.313+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8544005020 con 0x7f854c102290 2024-03-18T15:58:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.313+0000 7f853effd640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 0x7f85380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.314+0000 7f8551fbc640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 0x7f85380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.314+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f85440ceff0 con 0x7f854c102290 2024-03-18T15:58:00.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.314+0000 7f8551fbc640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 0x7f85380a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f8540005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:00.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.318+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f854409c1e0 con 0x7f854c102290 2024-03-18T15:58:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:58:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T15:58:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:58:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:58:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-18T15:58:00.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-18T15:58:00.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: pgmap v288: 97 pgs: 25 creating+peering, 46 active+clean, 26 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-18T15:58:00.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:00 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:58:00.493 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:58:00.493 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T15:58:00.493 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:58:00.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:58:00.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-18T15:58:00.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-18T15:58:00.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: pgmap v288: 97 pgs: 25 creating+peering, 46 active+clean, 26 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-18T15:58:00.494 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:00 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/204199161' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: pgmap v288: 97 pgs: 25 creating+peering, 46 active+clean, 26 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-18T15:58:00.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:00 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T15:58:00.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:00.532+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f854c02d390 con 0x7f854c102290 2024-03-18T15:58:01.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.190+0000 7f853effd640 1 -- 172.21.15.150:0/2097298708 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7f854409bae0 con 0x7f854c102290 2024-03-18T15:58:01.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 msgr2=0x7f85380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 0x7f85380a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8540000c00 tx=0x7f8540005fd0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 msgr2=0x7f854c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f854400eb10 tx=0x7f854400c2d0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 shutdown_connections 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f85380a2800 0x7f85380a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f854c102290 0x7f854c19f0f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f854c1018d0 0x7f854c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 --2- 172.21.15.150:0/2097298708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f854c1006e0 0x7f854c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 >> 172.21.15.150:0/2097298708 conn(0x7f854c0fbc70 msgr2=0x7f854c104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 shutdown_connections 2024-03-18T15:58:01.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.192+0000 7f8554220640 1 -- 172.21.15.150:0/2097298708 wait complete. 2024-03-18T15:58:01.207 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-18T15:58:01.207 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 -- 172.21.15.150:0/493468455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd01018d0 msgr2=0x7f0dd0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 --2- 172.21.15.150:0/493468455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd01018d0 0x7f0dd0101d50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009900 tx=0x7f0db802bc30 comp rx=0 tx=0).stop 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 -- 172.21.15.150:0/493468455 shutdown_connections 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 --2- 172.21.15.150:0/493468455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd0102290 0x7f0dd0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 --2- 172.21.15.150:0/493468455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd01018d0 0x7f0dd0101d50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 --2- 172.21.15.150:0/493468455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.314+0000 7f0dd6297640 1 -- 172.21.15.150:0/493468455 >> 172.21.15.150:0/493468455 conn(0x7f0dd00fbc70 msgr2=0x7f0dd00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- 172.21.15.150:0/493468455 shutdown_connections 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- 172.21.15.150:0/493468455 wait complete. 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 Processor -- start 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- start start 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 0x7f0dd019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 0x7f0dd019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd010c360 con 0x7f0dd01006e0 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd010c1e0 con 0x7f0dd0102290 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd6297640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd010c4e0 con 0x7f0dd01018d0 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60800/0 (socket says 172.21.15.150:60800) 2024-03-18T15:58:01.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dcffff640 1 -- 172.21.15.150:0/858410042 learned_addr learned my addr 172.21.15.150:0/858410042 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dc7fff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 0x7f0dd019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.315+0000 7f0dd4834640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 0x7f0dd019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 -- 172.21.15.150:0/858410042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 msgr2=0x7f0dd019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 0x7f0dd019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 -- 172.21.15.150:0/858410042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 msgr2=0x7f0dd019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 0x7f0dd019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 -- 172.21.15.150:0/858410042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd019f740 con 0x7f0dd01006e0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dc7fff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 0x7f0dd019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dd4834640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 0x7f0dd019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcffff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000e9c0 tx=0x7f0dc000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc000cde0 con 0x7f0dd01006e0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd019fa30 con 0x7f0dd01006e0 2024-03-18T15:58:01.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dc0010910 con 0x7f0dd01006e0 2024-03-18T15:58:01.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd019ffa0 con 0x7f0dd01006e0 2024-03-18T15:58:01.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.316+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00049e0 con 0x7f0dd01006e0 2024-03-18T15:58:01.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.317+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd0107ad0 con 0x7f0dd01006e0 2024-03-18T15:58:01.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.318+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0dc001f020 con 0x7f0dd01006e0 2024-03-18T15:58:01.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.319+0000 7f0dcdffb640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 0x7f0dbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.319+0000 7f0dc7fff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 0x7f0dbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.319+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0dc00cf0d0 con 0x7f0dd01006e0 2024-03-18T15:58:01.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.320+0000 7f0dc7fff640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 0x7f0dbc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0db802c7b0 tx=0x7f0db8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.323+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0dc0026070 con 0x7f0dd01006e0 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:active 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: Cluster is now healthy 2024-03-18T15:58:01.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-18T15:58:01.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:58:01.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:58:01.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:01 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:58:01.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:58:01.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:active 2024-03-18T15:58:01.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:58:01.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:58:01.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: Cluster is now healthy 2024-03-18T15:58:01.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-18T15:58:01.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:58:01.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:58:01.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:01 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:58:01.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.528+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0dd0061c20 con 0x7f0dbc0a2800 2024-03-18T15:58:01.535 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1192243283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T15:58:01.535 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] up:active 2024-03-18T15:58:01.535 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: Cluster is now healthy 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T15:58:01.536 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T15:58:01.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.546+0000 7f0dcdffb640 1 -- 172.21.15.150:0/858410042 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0dd0061c20 con 0x7f0dbc0a2800 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 msgr2=0x7f0dbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 0x7f0dbc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0db802c7b0 tx=0x7f0db8005fd0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 msgr2=0x7f0dd0069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000e9c0 tx=0x7f0dc000ee80 comp rx=0 tx=0).stop 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 shutdown_connections 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0dbc0a2800 0x7f0dbc0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0dd0102290 0x7f0dd019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0dd01018d0 0x7f0dd019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 --2- 172.21.15.150:0/858410042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dd01006e0 0x7f0dd0069510 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 >> 172.21.15.150:0/858410042 conn(0x7f0dd00fbc70 msgr2=0x7f0dd0104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 shutdown_connections 2024-03-18T15:58:01.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.548+0000 7f0dd6297640 1 -- 172.21.15.150:0/858410042 wait complete. 2024-03-18T15:58:01.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 -- 172.21.15.150:0/3272896134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 msgr2=0x7fa72c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 --2- 172.21.15.150:0/3272896134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c100b00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009860 tx=0x7fa71c02bb70 comp rx=0 tx=0).stop 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 -- 172.21.15.150:0/3272896134 shutdown_connections 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 --2- 172.21.15.150:0/3272896134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c102290 0x7fa72c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 --2- 172.21.15.150:0/3272896134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c1018d0 0x7fa72c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 --2- 172.21.15.150:0/3272896134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c100b00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 -- 172.21.15.150:0/3272896134 >> 172.21.15.150:0/3272896134 conn(0x7fa72c0fbc70 msgr2=0x7fa72c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 -- 172.21.15.150:0/3272896134 shutdown_connections 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.669+0000 7fa7327f3640 1 -- 172.21.15.150:0/3272896134 wait complete. 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 Processor -- start 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 -- start start 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 0x7fa72c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 0x7fa72c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c10c360 con 0x7fa72c102290 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c10c1e0 con 0x7fa72c1006e0 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa7327f3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c10c4e0 con 0x7fa72c1018d0 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa72bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa730d90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 0x7fa72c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa72b7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 0x7fa72c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa72bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:47478/0 (socket says 172.21.15.150:47478) 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.670+0000 7fa72bfff640 1 -- 172.21.15.150:0/1759921595 learned_addr learned my addr 172.21.15.150:0/1759921595 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 -- 172.21.15.150:0/1759921595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 msgr2=0x7fa72c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 0x7fa72c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 -- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 msgr2=0x7fa72c19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 0x7fa72c19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 -- 172.21.15.150:0/1759921595 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c19f7b0 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72b7fe640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 0x7fa72c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa730d90640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 0x7fa72c19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa72bfff640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa71c02c0b0 tx=0x7fa71c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c041020 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71c02c9f0 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c19f9e0 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c039040 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.671+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c19fed0 con 0x7fa72c1006e0 2024-03-18T15:58:01.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.672+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c107aa0 con 0x7fa72c1006e0 2024-03-18T15:58:01.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.673+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa71c044020 con 0x7fa72c1006e0 2024-03-18T15:58:01.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.673+0000 7fa7297fa640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 0x7fa7140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.674+0000 7fa72b7fe640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 0x7fa7140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.674+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa71c0f0610 con 0x7fa72c1006e0 2024-03-18T15:58:01.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.674+0000 7fa72b7fe640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 0x7fa7140a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7200096c0 tx=0x7fa720009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.677+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa71c04b020 con 0x7fa72c1006e0 2024-03-18T15:58:01.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:58:01.533+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:58:01.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:58:01.533+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:58:01.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:58:01.533+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:58:01.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:58:01.533+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:58:01.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:58:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:58:01.533+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:58:01.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.883+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa72c061cb0 con 0x7fa7140a2800 2024-03-18T15:58:01.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.906+0000 7fa7297fa640 1 -- 172.21.15.150:0/1759921595 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa72c061cb0 con 0x7fa7140a2800 2024-03-18T15:58:01.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 msgr2=0x7fa7140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 0x7fa7140a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7200096c0 tx=0x7fa720009260 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 msgr2=0x7fa72c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa71c02c0b0 tx=0x7fa71c032430 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 shutdown_connections 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa7140a2800 0x7fa7140a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72c102290 0x7fa72c19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa72c1018d0 0x7fa72c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 --2- 172.21.15.150:0/1759921595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa72c1006e0 0x7fa72c0694b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 >> 172.21.15.150:0/1759921595 conn(0x7fa72c0fbc70 msgr2=0x7fa72c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 shutdown_connections 2024-03-18T15:58:01.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:01.908+0000 7fa7327f3640 1 -- 172.21.15.150:0/1759921595 wait complete. 2024-03-18T15:58:01.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T15:58:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 -- 172.21.15.150:0/3436952955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9241022c0 msgr2=0x7fb9241046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 --2- 172.21.15.150:0/3436952955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9241022c0 0x7fb9241046b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb914009900 tx=0x7fb91402bc30 comp rx=0 tx=0).stop 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 -- 172.21.15.150:0/3436952955 shutdown_connections 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 --2- 172.21.15.150:0/3436952955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924107520 0x7fb924109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 --2- 172.21.15.150:0/3436952955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb924104bf0 0x7fb924106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 --2- 172.21.15.150:0/3436952955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9241022c0 0x7fb9241046b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 -- 172.21.15.150:0/3436952955 >> 172.21.15.150:0/3436952955 conn(0x7fb9240fbc70 msgr2=0x7fb9240fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 -- 172.21.15.150:0/3436952955 shutdown_connections 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.027+0000 7fb9290a1640 1 -- 172.21.15.150:0/3436952955 wait complete. 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 Processor -- start 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 -- start start 2024-03-18T15:58:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9241022c0 0x7fb924197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924104bf0 0x7fb9241980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb92410d480 con 0x7fb924107520 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb92410d300 con 0x7fb924104bf0 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb9290a1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb92410d600 con 0x7fb9241022c0 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60838/0 (socket says 172.21.15.150:60838) 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 -- 172.21.15.150:0/4052524904 learned_addr learned my addr 172.21.15.150:0/4052524904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:02.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb922d9d640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9241022c0 0x7fb924197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92259c640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924104bf0 0x7fb9241980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 -- 172.21.15.150:0/4052524904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9241022c0 msgr2=0x7fb924197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9241022c0 0x7fb924197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 -- 172.21.15.150:0/4052524904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924104bf0 msgr2=0x7fb9241980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924104bf0 0x7fb9241980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.028+0000 7fb92359e640 1 -- 172.21.15.150:0/4052524904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92419f7a0 con 0x7fb924107520 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb92359e640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00eb10 tx=0x7fb91c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c004b20 con 0x7fb924107520 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92419fa90 con 0x7fb924107520 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91c004cc0 con 0x7fb924107520 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c010760 con 0x7fb924107520 2024-03-18T15:58:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.029+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92419ffe0 con 0x7fb924107520 2024-03-18T15:58:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.030+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9240fd760 con 0x7fb924107520 2024-03-18T15:58:02.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.031+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb91c005020 con 0x7fb924107520 2024-03-18T15:58:02.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.032+0000 7fb90bfff640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 0x7fb90c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.032+0000 7fb922d9d640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 0x7fb90c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.032+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb91c0cee80 con 0x7fb924107520 2024-03-18T15:58:02.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.032+0000 7fb922d9d640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 0x7fb90c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb91402c7b0 tx=0x7fb914005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.036+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb91c09c070 con 0x7fb924107520 2024-03-18T15:58:02.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.211+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb91c097fe0 con 0x7fb924107520 2024-03-18T15:58:02.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.248+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fb924061c20 con 0x7fb90c0a2800 2024-03-18T15:58:02.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.252+0000 7fb90bfff640 1 -- 172.21.15.150:0/4052524904 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb924061c20 con 0x7fb90c0a2800 2024-03-18T15:58:02.253 INFO:teuthology.orchestra.run.smithi150.stdout:/volumes/qa/sv_0/869a4a06-3834-4f24-b453-1f90f13c7476 2024-03-18T15:58:02.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 msgr2=0x7fb90c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 0x7fb90c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb91402c7b0 tx=0x7fb914005fd0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 msgr2=0x7fb92419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00eb10 tx=0x7fb91c00c2d0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 shutdown_connections 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb90c0a2800 0x7fb90c0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb924107520 0x7fb92419f130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb924104bf0 0x7fb9241980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 --2- 172.21.15.150:0/4052524904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9241022c0 0x7fb924197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 >> 172.21.15.150:0/4052524904 conn(0x7fb9240fbc70 msgr2=0x7fb924102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 shutdown_connections 2024-03-18T15:58:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.254+0000 7fb9290a1640 1 -- 172.21.15.150:0/4052524904 wait complete. 2024-03-18T15:58:02.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T15:58:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 -- 172.21.15.150:0/847577821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f4102a90 msgr2=0x7f41f4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 --2- 172.21.15.150:0/847577821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f4102a90 0x7f41f4102f10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009930 tx=0x7f41e002bc30 comp rx=0 tx=0).stop 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 -- 172.21.15.150:0/847577821 shutdown_connections 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 --2- 172.21.15.150:0/847577821 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4103450 0x7f41f4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 --2- 172.21.15.150:0/847577821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f4102a90 0x7f41f4102f10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 --2- 172.21.15.150:0/847577821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f40ffac0 0x7f41f40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 -- 172.21.15.150:0/847577821 >> 172.21.15.150:0/847577821 conn(0x7f41f40fb3f0 msgr2=0x7f41f40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 -- 172.21.15.150:0/847577821 shutdown_connections 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 -- 172.21.15.150:0/847577821 wait complete. 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.371+0000 7f41faf10640 1 Processor -- start 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 -- start start 2024-03-18T15:58:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4102a90 0x7f41f4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f4103450 0x7f41f419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41f410d480 con 0x7f41f40ffac0 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41f410d300 con 0x7f41f4103450 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41faf10640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f41f410d600 con 0x7f41f4102a90 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60854/0 (socket says 172.21.15.150:60854) 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 -- 172.21.15.150:0/1645968646 learned_addr learned my addr 172.21.15.150:0/1645968646 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f3fff640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4102a90 0x7f41f4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f94ad640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f4103450 0x7f41f419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 -- 172.21.15.150:0/1645968646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4102a90 msgr2=0x7f41f4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4102a90 0x7f41f4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 -- 172.21.15.150:0/1645968646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f4103450 msgr2=0x7f41f419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f4103450 0x7f41f419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 -- 172.21.15.150:0/1645968646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f419f770 con 0x7f41f40ffac0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.372+0000 7f41f8cac640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f41e800e9c0 tx=0x7f41e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.373+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e800cd80 con 0x7f41f40ffac0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.373+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e80035f0 con 0x7f41f40ffac0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.373+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8010690 con 0x7f41f40ffac0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.373+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f419fa60 con 0x7f41f40ffac0 2024-03-18T15:58:02.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.373+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f419ffd0 con 0x7f41f40ffac0 2024-03-18T15:58:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.374+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f4108ce0 con 0x7f41f40ffac0 2024-03-18T15:58:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.375+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f41e80037a0 con 0x7f41f40ffac0 2024-03-18T15:58:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.375+0000 7f41f1ffb640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 0x7f41e40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.375+0000 7f41f3fff640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 0x7f41e40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.375+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f41e80ced70 con 0x7f41f40ffac0 2024-03-18T15:58:02.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.376+0000 7f41f3fff640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 0x7f41e40a4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41e002efd0 tx=0x7f41e0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.380+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41e809be60 con 0x7f41f40ffac0 2024-03-18T15:58:02.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:58:02.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:active 2024-03-18T15:58:02.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:58:02.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:58:02.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: pgmap v289: 97 pgs: 14 creating+peering, 83 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-03-18T15:58:02.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-mon[36784]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:02.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:58:02.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:active 2024-03-18T15:58:02.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:58:02.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:58:02.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: pgmap v289: 97 pgs: 14 creating+peering, 83 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-03-18T15:58:02.519 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:02 smithi156 ceph-mon[35919]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:02.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.583+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f41f4061c20 con 0x7f41e40a2910 2024-03-18T15:58:02.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2097298708' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T15:58:02.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] up:active 2024-03-18T15:58:02.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-03-18T15:58:02.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:58:02.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: pgmap v289: 97 pgs: 14 creating+peering, 83 active+clean; 581 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 4.5 KiB/s wr, 13 op/s 2024-03-18T15:58:02.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:02 smithi150 ceph-mon[31946]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.604+0000 7f41f1ffb640 1 -- 172.21.15.150:0/1645968646 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f41f4061c20 con 0x7f41e40a2910 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 msgr2=0x7f41e40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 0x7f41e40a4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f41e002efd0 tx=0x7f41e0000bc0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 msgr2=0x7f41f4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f41e800e9c0 tx=0x7f41e800ee80 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 shutdown_connections 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f41e40a2910 0x7f41e40a4dd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41f4103450 0x7f41f419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41f4102a90 0x7f41f4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 --2- 172.21.15.150:0/1645968646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41f40ffac0 0x7f41f4197b50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 >> 172.21.15.150:0/1645968646 conn(0x7f41f40fb3f0 msgr2=0x7f41f4105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 shutdown_connections 2024-03-18T15:58:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.606+0000 7f41faf10640 1 -- 172.21.15.150:0/1645968646 wait complete. 2024-03-18T15:58:02.619 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- 172.21.15.150:0/4012987044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741018f0 msgr2=0x7f6774101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- 172.21.15.150:0/4012987044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741018f0 0x7f6774101d70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6770009900 tx=0x7f677002bc30 comp rx=0 tx=0).stop 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- 172.21.15.150:0/4012987044 shutdown_connections 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- 172.21.15.150:0/4012987044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741022b0 0x7f6774108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- 172.21.15.150:0/4012987044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741018f0 0x7f6774101d70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- 172.21.15.150:0/4012987044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 0x7f6774100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- 172.21.15.150:0/4012987044 >> 172.21.15.150:0/4012987044 conn(0x7f67740fbc90 msgr2=0x7f67740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- 172.21.15.150:0/4012987044 shutdown_connections 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- 172.21.15.150:0/4012987044 wait complete. 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 Processor -- start 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- start start 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 0x7f6774197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 0x7f6774198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.725+0000 7f677c281640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f677410d4a0 con 0x7f67741022b0 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677c281640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f677410d320 con 0x7f67741018f0 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677c281640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f677410d620 con 0x7f6774100700 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54988/0 (socket says 172.21.15.150:54988) 2024-03-18T15:58:02.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 -- 172.21.15.150:0/2160591343 learned_addr learned my addr 172.21.15.150:0/2160591343 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a01d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 0x7f6774197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677981c640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 0x7f6774198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 -- 172.21.15.150:0/2160591343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 msgr2=0x7f6774197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 0x7f6774197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 -- 172.21.15.150:0/2160591343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 msgr2=0x7f6774198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 0x7f6774198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 -- 172.21.15.150:0/2160591343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677419f730 con 0x7f67741022b0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677981c640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 0x7f6774198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677a81e640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f676c00eb10 tx=0x7f676c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c004b20 con 0x7f67741022b0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677419fa20 con 0x7f67741022b0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f676c004cc0 con 0x7f67741022b0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010760 con 0x7f67741022b0 2024-03-18T15:58:02.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.726+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677419fef0 con 0x7f67741022b0 2024-03-18T15:58:02.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.727+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6774107a20 con 0x7f67741022b0 2024-03-18T15:58:02.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.728+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f676c005020 con 0x7f67741022b0 2024-03-18T15:58:02.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.729+0000 7f676b7fe640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 0x7f67600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:02.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.729+0000 7f677a01d640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 0x7f67600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:02.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.729+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f676c0cf060 con 0x7f67741022b0 2024-03-18T15:58:02.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.729+0000 7f677a01d640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 0x7f67600a4d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6774101750 tx=0x7f6764005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:02.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.734+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f676c09c150 con 0x7f67741022b0 2024-03-18T15:58:02.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.939+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6774061c20 con 0x7f67600a2890 2024-03-18T15:58:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.943+0000 7f676b7fe640 1 -- 172.21.15.150:0/2160591343 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6774061c20 con 0x7f67600a2890 2024-03-18T15:58:02.944 INFO:teuthology.orchestra.run.smithi150.stdout:/volumes/qa/sv_1/3b1f2cfb-cb65-41f3-9829-04ee269f3064 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 msgr2=0x7f67600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 0x7f67600a4d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6774101750 tx=0x7f6764005f90 comp rx=0 tx=0).stop 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 msgr2=0x7f677419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f676c00eb10 tx=0x7f676c00c2d0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 shutdown_connections 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67600a2890 0x7f67600a4d50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67741022b0 0x7f677419f0c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67741018f0 0x7f6774198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 --2- 172.21.15.150:0/2160591343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6774100700 0x7f6774197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 >> 172.21.15.150:0/2160591343 conn(0x7f67740fbc90 msgr2=0x7f6774104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 shutdown_connections 2024-03-18T15:58:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:02.945+0000 7f677c281640 1 -- 172.21.15.150:0/2160591343 wait complete. 2024-03-18T15:58:02.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:58:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.057+0000 7f9a2eac6640 1 -- 172.21.15.150:0/1180795173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a281022b0 msgr2=0x7f9a28108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.057+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/1180795173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a281022b0 0x7f9a28108b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009be0 tx=0x7f9a2002bef0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- 172.21.15.150:0/1180795173 shutdown_connections 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/1180795173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a281022b0 0x7f9a28108b40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/1180795173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a28101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/1180795173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a28100700 0x7f9a28100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- 172.21.15.150:0/1180795173 >> 172.21.15.150:0/1180795173 conn(0x7f9a280fbc70 msgr2=0x7f9a280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- 172.21.15.150:0/1180795173 shutdown_connections 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- 172.21.15.150:0/1180795173 wait complete. 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 Processor -- start 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- start start 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a28100700 0x7f9a28069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a281022b0 0x7f9a2819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810c380 con 0x7f9a281018f0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810c200 con 0x7f9a281022b0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2eac6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810c500 con 0x7f9a28100700 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a27fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2c862640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a28100700 0x7f9a28069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a27fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55000/0 (socket says 172.21.15.150:55000) 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a27fff640 1 -- 172.21.15.150:0/779490158 learned_addr learned my addr 172.21.15.150:0/779490158 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.058+0000 7f9a2d063640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a281022b0 0x7f9a2819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 -- 172.21.15.150:0/779490158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a28100700 msgr2=0x7f9a28069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a28100700 0x7f9a28069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 -- 172.21.15.150:0/779490158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a281022b0 msgr2=0x7f9a2819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a281022b0 0x7f9a2819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 -- 172.21.15.150:0/779490158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2819f6b0 con 0x7f9a281018f0 2024-03-18T15:58:03.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a27fff640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400e9c0 tx=0x7f9a1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1400ccc0 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a14005500 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2819f9a0 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14010430 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.059+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2819fef0 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.060+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a281079c0 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.061+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9a140046c0 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.061+0000 7f9a25ffb640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 0x7f9a180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.062+0000 7f9a2c862640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 0x7f9a180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.062+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9a140ced50 con 0x7f9a281018f0 2024-03-18T15:58:03.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.062+0000 7f9a2c862640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 0x7f9a180a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a28101750 tx=0x7f9a1c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.066+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a1409be40 con 0x7f9a281018f0 2024-03-18T15:58:03.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.278+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a2802d1b0 con 0x7f9a281018f0 2024-03-18T15:58:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.279+0000 7f9a25ffb640 1 -- 172.21.15.150:0/779490158 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f9a1409b740 con 0x7f9a281018f0 2024-03-18T15:58:03.281 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:03.281 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:58:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:58:03.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 msgr2=0x7f9a180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 0x7f9a180a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a28101750 tx=0x7f9a1c005f90 comp rx=0 tx=0).stop 2024-03-18T15:58:03.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 msgr2=0x7f9a2819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400e9c0 tx=0x7f9a1400ee80 comp rx=0 tx=0).stop 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 shutdown_connections 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9a180a2890 0x7f9a180a4d50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a281022b0 0x7f9a2819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a281018f0 0x7f9a2819c780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 --2- 172.21.15.150:0/779490158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a28100700 0x7f9a28069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 >> 172.21.15.150:0/779490158 conn(0x7f9a280fbc70 msgr2=0x7f9a28104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 shutdown_connections 2024-03-18T15:58:03.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.281+0000 7f9a2eac6640 1 -- 172.21.15.150:0/779490158 wait complete. 2024-03-18T15:58:03.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:58:03.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.394+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/144068768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1022b0 msgr2=0x7fd2cc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.394+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/144068768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc108b40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4009ca0 tx=0x7fd2c402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/144068768 shutdown_connections 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/144068768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc108b40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/144068768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/144068768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc100700 0x7fd2cc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/144068768 >> 172.21.15.150:0/144068768 conn(0x7fd2cc0fbc70 msgr2=0x7fd2cc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/144068768 shutdown_connections 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/144068768 wait complete. 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 Processor -- start 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- start start 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc100700 0x7fd2cc197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc10d460 con 0x7fd2cc1018f0 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc10d2e0 con 0x7fd2cc100700 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d3e9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc10d5e0 con 0x7fd2cc1022b0 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d1435640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d1435640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55006/0 (socket says 172.21.15.150:55006) 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d1435640 1 -- 172.21.15.150:0/2329708125 learned_addr learned my addr 172.21.15.150:0/2329708125 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d1c36640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc100700 0x7fd2cc197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.395+0000 7fd2d2437640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 -- 172.21.15.150:0/2329708125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc1022b0 msgr2=0x7fd2cc19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 -- 172.21.15.150:0/2329708125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc100700 msgr2=0x7fd2cc197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc100700 0x7fd2cc197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 -- 172.21.15.150:0/2329708125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc19f6c0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d1435640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800e9f0 tx=0x7fd2c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800cd80 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc19f9b0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2c8010910 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80188f0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.396+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc19ff50 con 0x7fd2cc1018f0 2024-03-18T15:58:03.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.397+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2cc1079c0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.398+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd2c8010ac0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.399+0000 7fd2beffd640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 0x7fd2b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.399+0000 7fd2d1c36640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 0x7fd2b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.399+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd2c80cedf0 con 0x7fd2cc1018f0 2024-03-18T15:58:03.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.399+0000 7fd2d1c36640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 0x7fd2b80a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc101750 tx=0x7fd2c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.403+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2c809b890 con 0x7fd2cc1018f0 2024-03-18T15:58:03.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:03 smithi191 ceph-mon[36784]: from='client.24554 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:03 smithi191 ceph-mon[36784]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T15:58:03.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:03 smithi191 ceph-mon[36784]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:03 smithi191 ceph-mon[36784]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.465 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: 2024-03-18T15:58:02.994+0000 7f49113f8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T15:58:03.465 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: 2024-03-18T15:58:02.994+0000 7f49113f8700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-18T15:58:03.465 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 15:58:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: 2024-03-18T15:58:02.994+0000 7f49113f8700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-18T15:58:03.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:03 smithi156 ceph-mon[35919]: from='client.24554 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:03 smithi156 ceph-mon[35919]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T15:58:03.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:03 smithi156 ceph-mon[35919]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:03 smithi156 ceph-mon[35919]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.579+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2cc02d360 con 0x7fd2cc1018f0 2024-03-18T15:58:03.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.581+0000 7fd2beffd640 1 -- 172.21.15.150:0/2329708125 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22275 (secure 0 0 0) 0x7fd2c8084e20 con 0x7fd2cc1018f0 2024-03-18T15:58:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:03.582 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:58.165838+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 msgr2=0x7fd2b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 0x7fd2b80a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc101750 tx=0x7fd2c0005f90 comp rx=0 tx=0).stop 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 msgr2=0x7fd2cc197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800e9f0 tx=0x7fd2c800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 shutdown_connections 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd2b80a2890 0x7fd2b80a4d50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2cc1022b0 0x7fd2cc19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc1018f0 0x7fd2cc197fe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 --2- 172.21.15.150:0/2329708125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2cc100700 0x7fd2cc197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 >> 172.21.15.150:0/2329708125 conn(0x7fd2cc0fbc70 msgr2=0x7fd2cc104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 shutdown_connections 2024-03-18T15:58:03.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.583+0000 7fd2d3e9a640 1 -- 172.21.15.150:0/2329708125 wait complete. 2024-03-18T15:58:03.591 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:03 smithi150 ceph-mon[31946]: from='client.24554 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:03 smithi150 ceph-mon[31946]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T15:58:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:03 smithi150 ceph-mon[31946]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:03 smithi150 ceph-mon[31946]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:03.595 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T15:58:03.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.695+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/1793041645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 msgr2=0x7f1a64100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.695+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/1793041645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a64100b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009ca0 tx=0x7f1a5c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/1793041645 shutdown_connections 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/1793041645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641022b0 0x7f1a64108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/1793041645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641018f0 0x7f1a64101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/1793041645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a64100b20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/1793041645 >> 172.21.15.150:0/1793041645 conn(0x7f1a640fbc70 msgr2=0x7f1a640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/1793041645 shutdown_connections 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/1793041645 wait complete. 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 Processor -- start 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- start start 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641018f0 0x7f1a6419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641022b0 0x7f1a6419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410c350 con 0x7f1a64100700 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410c1d0 con 0x7f1a641018f0 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6bcd1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6410c4d0 con 0x7f1a641022b0 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a69a6d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a69a6d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55024/0 (socket says 172.21.15.150:55024) 2024-03-18T15:58:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a69a6d640 1 -- 172.21.15.150:0/2938029814 learned_addr learned my addr 172.21.15.150:0/2938029814 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.696+0000 7f1a6926c640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641018f0 0x7f1a6419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a6a26e640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641022b0 0x7f1a6419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 -- 172.21.15.150:0/2938029814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641022b0 msgr2=0x7f1a6419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641022b0 0x7f1a6419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 -- 172.21.15.150:0/2938029814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641018f0 msgr2=0x7f1a6419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641018f0 0x7f1a6419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 -- 172.21.15.150:0/2938029814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6419f740 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a69a6d640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0324c0 tx=0x7f1a5c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c041020 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a5c034cd0 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c039040 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6419f9d0 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.697+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6419ff10 con 0x7f1a64100700 2024-03-18T15:58:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.698+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a64107a20 con 0x7f1a64100700 2024-03-18T15:58:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.699+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1a5c044020 con 0x7f1a64100700 2024-03-18T15:58:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.699+0000 7f1a62ffd640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 0x7f1a4c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.700+0000 7f1a6926c640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 0x7f1a4c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.700+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1a5c0f0760 con 0x7f1a64100700 2024-03-18T15:58:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.700+0000 7f1a6926c640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 0x7f1a4c0a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1a580095f0 tx=0x7f1a58005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.704+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a5c04b020 con 0x7f1a64100700 2024-03-18T15:58:03.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.874+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a64061c20 con 0x7f1a4c0a2910 2024-03-18T15:58:03.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.882+0000 7f1a62ffd640 1 -- 172.21.15.150:0/2938029814 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258916 (secure 0 0 0) 0x7f1a64061c20 con 0x7f1a4c0a2910 2024-03-18T15:58:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:03.888 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-18T15:58:03.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 msgr2=0x7f1a4c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 0x7f1a4c0a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1a580095f0 tx=0x7f1a58005f90 comp rx=0 tx=0).stop 2024-03-18T15:58:03.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 msgr2=0x7f1a640694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0324c0 tx=0x7f1a5c032f40 comp rx=0 tx=0).stop 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 shutdown_connections 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1a4c0a2910 0x7f1a4c0a4dd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1a641022b0 0x7f1a6419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a641018f0 0x7f1a6419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 --2- 172.21.15.150:0/2938029814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64100700 0x7f1a640694d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 >> 172.21.15.150:0/2938029814 conn(0x7f1a640fbc70 msgr2=0x7f1a64104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 shutdown_connections 2024-03-18T15:58:03.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:03.885+0000 7f1a6bcd1640 1 -- 172.21.15.150:0/2938029814 wait complete. 2024-03-18T15:58:03.898 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-18T15:58:03.444367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595478,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":144,"num_read_kb":126,"num_write":286,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377004,"kb_used_data":11172,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564820,"statfs":{"total":1151940427776,"available":1151554375680,"internally_reserved":0,"allocated":11440128,"data_stored":7084258,"data_compressed":108786,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5110,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.288398"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169834+0000","last_change":"2024-03-18T15:57:57.181734+0000","last_active":"2024-03-18T15:57:59.169834+0000","last_peered":"2024-03-18T15:57:59.169834+0000","last_clean":"2024-03-18T15:57:59.169834+0000","last_became_active":"2024-03-18T15:57:57.181321+0000","last_became_peered":"2024-03-18T15:57:57.181321+0000","last_unstale":"2024-03-18T15:57:59.169834+0000","last_undegraded":"2024-03-18T15:57:59.169834+0000","last_fullsized":"2024-03-18T15:57:59.169834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:14.921877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170588+0000","last_change":"2024-03-18T15:57:57.175040+0000","last_active":"2024-03-18T15:57:59.170588+0000","last_peered":"2024-03-18T15:57:59.170588+0000","last_clean":"2024-03-18T15:57:59.170588+0000","last_became_active":"2024-03-18T15:57:57.174721+0000","last_became_peered":"2024-03-18T15:57:57.174721+0000","last_unstale":"2024-03-18T15:57:59.170588+0000","last_undegraded":"2024-03-18T15:57:59.170588+0000","last_fullsized":"2024-03-18T15:57:59.170588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:30.844693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170468+0000","last_change":"2024-03-18T15:57:57.183409+0000","last_active":"2024-03-18T15:57:59.170468+0000","last_peered":"2024-03-18T15:57:59.170468+0000","last_clean":"2024-03-18T15:57:59.170468+0000","last_became_active":"2024-03-18T15:57:57.183089+0000","last_became_peered":"2024-03-18T15:57:57.183089+0000","last_unstale":"2024-03-18T15:57:59.170468+0000","last_undegraded":"2024-03-18T15:57:59.170468+0000","last_fullsized":"2024-03-18T15:57:59.170468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:25.702402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171710+0000","last_change":"2024-03-18T15:57:57.183104+0000","last_active":"2024-03-18T15:57:59.171710+0000","last_peered":"2024-03-18T15:57:59.171710+0000","last_clean":"2024-03-18T15:57:59.171710+0000","last_became_active":"2024-03-18T15:57:57.182750+0000","last_became_peered":"2024-03-18T15:57:57.182750+0000","last_unstale":"2024-03-18T15:57:59.171710+0000","last_undegraded":"2024-03-18T15:57:59.171710+0000","last_fullsized":"2024-03-18T15:57:59.171710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:25.369309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167428+0000","last_change":"2024-03-18T15:57:57.195272+0000","last_active":"2024-03-18T15:57:59.167428+0000","last_peered":"2024-03-18T15:57:59.167428+0000","last_clean":"2024-03-18T15:57:59.167428+0000","last_became_active":"2024-03-18T15:57:57.192605+0000","last_became_peered":"2024-03-18T15:57:57.192605+0000","last_unstale":"2024-03-18T15:57:59.167428+0000","last_undegraded":"2024-03-18T15:57:59.167428+0000","last_fullsized":"2024-03-18T15:57:59.167428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:05.719973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170423+0000","last_change":"2024-03-18T15:57:57.177918+0000","last_active":"2024-03-18T15:57:59.170423+0000","last_peered":"2024-03-18T15:57:59.170423+0000","last_clean":"2024-03-18T15:57:59.170423+0000","last_became_active":"2024-03-18T15:57:57.177551+0000","last_became_peered":"2024-03-18T15:57:57.177551+0000","last_unstale":"2024-03-18T15:57:59.170423+0000","last_undegraded":"2024-03-18T15:57:59.170423+0000","last_fullsized":"2024-03-18T15:57:59.170423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:15.033755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170101+0000","last_change":"2024-03-18T15:57:57.179306+0000","last_active":"2024-03-18T15:57:59.170101+0000","last_peered":"2024-03-18T15:57:59.170101+0000","last_clean":"2024-03-18T15:57:59.170101+0000","last_became_active":"2024-03-18T15:57:57.178955+0000","last_became_peered":"2024-03-18T15:57:57.178955+0000","last_unstale":"2024-03-18T15:57:59.170101+0000","last_undegraded":"2024-03-18T15:57:59.170101+0000","last_fullsized":"2024-03-18T15:57:59.170101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:47.343910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.172976+0000","last_change":"2024-03-18T15:57:57.182548+0000","last_active":"2024-03-18T15:57:58.172976+0000","last_peered":"2024-03-18T15:57:58.172976+0000","last_clean":"2024-03-18T15:57:58.172976+0000","last_became_active":"2024-03-18T15:57:57.180297+0000","last_became_peered":"2024-03-18T15:57:57.180297+0000","last_unstale":"2024-03-18T15:57:58.172976+0000","last_undegraded":"2024-03-18T15:57:58.172976+0000","last_fullsized":"2024-03-18T15:57:58.172976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:47.392449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174048+0000","last_change":"2024-03-18T15:57:57.190203+0000","last_active":"2024-03-18T15:57:58.174048+0000","last_peered":"2024-03-18T15:57:58.174048+0000","last_clean":"2024-03-18T15:57:58.174048+0000","last_became_active":"2024-03-18T15:57:57.189850+0000","last_became_peered":"2024-03-18T15:57:57.189850+0000","last_unstale":"2024-03-18T15:57:58.174048+0000","last_undegraded":"2024-03-18T15:57:58.174048+0000","last_fullsized":"2024-03-18T15:57:58.174048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:46.274479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171061+0000","last_change":"2024-03-18T15:57:57.171053+0000","last_active":"2024-03-18T15:57:59.171061+0000","last_peered":"2024-03-18T15:57:59.171061+0000","last_clean":"2024-03-18T15:57:59.171061+0000","last_became_active":"2024-03-18T15:57:57.170727+0000","last_became_peered":"2024-03-18T15:57:57.170727+0000","last_unstale":"2024-03-18T15:57:59.171061+0000","last_undegraded":"2024-03-18T15:57:59.171061+0000","last_fullsized":"2024-03-18T15:57:59.171061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170789+0000","last_change":"2024-03-18T15:57:57.184219+0000","last_active":"2024-03-18T15:57:59.170789+0000","last_peered":"2024-03-18T15:57:59.170789+0000","last_clean":"2024-03-18T15:57:59.170789+0000","last_became_active":"2024-03-18T15:57:57.183575+0000","last_became_peered":"2024-03-18T15:57:57.183575+0000","last_unstale":"2024-03-18T15:57:59.170789+0000","last_undegraded":"2024-03-18T15:57:59.170789+0000","last_fullsized":"2024-03-18T15:57:59.170789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:59.171634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171637+0000","last_change":"2024-03-18T15:57:57.185462+0000","last_active":"2024-03-18T15:57:59.171637+0000","last_peered":"2024-03-18T15:57:59.171637+0000","last_clean":"2024-03-18T15:57:59.171637+0000","last_became_active":"2024-03-18T15:57:57.185136+0000","last_became_peered":"2024-03-18T15:57:57.185136+0000","last_unstale":"2024-03-18T15:57:59.171637+0000","last_undegraded":"2024-03-18T15:57:59.171637+0000","last_fullsized":"2024-03-18T15:57:59.171637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:36.579297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169787+0000","last_change":"2024-03-18T15:57:57.176638+0000","last_active":"2024-03-18T15:57:59.169787+0000","last_peered":"2024-03-18T15:57:59.169787+0000","last_clean":"2024-03-18T15:57:59.169787+0000","last_became_active":"2024-03-18T15:57:57.176327+0000","last_became_peered":"2024-03-18T15:57:57.176327+0000","last_unstale":"2024-03-18T15:57:59.169787+0000","last_undegraded":"2024-03-18T15:57:59.169787+0000","last_fullsized":"2024-03-18T15:57:59.169787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:18.115306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170129+0000","last_change":"2024-03-18T15:57:57.172258+0000","last_active":"2024-03-18T15:57:59.170129+0000","last_peered":"2024-03-18T15:57:59.170129+0000","last_clean":"2024-03-18T15:57:59.170129+0000","last_became_active":"2024-03-18T15:57:57.171916+0000","last_became_peered":"2024-03-18T15:57:57.171916+0000","last_unstale":"2024-03-18T15:57:59.170129+0000","last_undegraded":"2024-03-18T15:57:59.170129+0000","last_fullsized":"2024-03-18T15:57:59.170129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:24.467000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169788+0000","last_change":"2024-03-18T15:57:57.173523+0000","last_active":"2024-03-18T15:57:59.169788+0000","last_peered":"2024-03-18T15:57:59.169788+0000","last_clean":"2024-03-18T15:57:59.169788+0000","last_became_active":"2024-03-18T15:57:57.173203+0000","last_became_peered":"2024-03-18T15:57:57.173203+0000","last_unstale":"2024-03-18T15:57:59.169788+0000","last_undegraded":"2024-03-18T15:57:59.169788+0000","last_fullsized":"2024-03-18T15:57:59.169788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:31.383153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173614+0000","last_change":"2024-03-18T15:57:57.185214+0000","last_active":"2024-03-18T15:57:58.173614+0000","last_peered":"2024-03-18T15:57:58.173614+0000","last_clean":"2024-03-18T15:57:58.173614+0000","last_became_active":"2024-03-18T15:57:57.183794+0000","last_became_peered":"2024-03-18T15:57:57.183794+0000","last_unstale":"2024-03-18T15:57:58.173614+0000","last_undegraded":"2024-03-18T15:57:58.173614+0000","last_fullsized":"2024-03-18T15:57:58.173614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:45.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169816+0000","last_change":"2024-03-18T15:57:57.183349+0000","last_active":"2024-03-18T15:57:59.169816+0000","last_peered":"2024-03-18T15:57:59.169816+0000","last_clean":"2024-03-18T15:57:59.169816+0000","last_became_active":"2024-03-18T15:57:57.182127+0000","last_became_peered":"2024-03-18T15:57:57.182127+0000","last_unstale":"2024-03-18T15:57:59.169816+0000","last_undegraded":"2024-03-18T15:57:59.169816+0000","last_fullsized":"2024-03-18T15:57:59.169816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:17.229945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594614+0000","last_change":"2024-03-18T15:57:57.182112+0000","last_active":"2024-03-18T15:57:58.594614+0000","last_peered":"2024-03-18T15:57:58.594614+0000","last_clean":"2024-03-18T15:57:58.594614+0000","last_became_active":"2024-03-18T15:57:57.181695+0000","last_became_peered":"2024-03-18T15:57:57.181695+0000","last_unstale":"2024-03-18T15:57:58.594614+0000","last_undegraded":"2024-03-18T15:57:58.594614+0000","last_fullsized":"2024-03-18T15:57:58.594614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:20.402597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350248+0000","last_change":"2024-03-18T15:57:57.182187+0000","last_active":"2024-03-18T15:57:58.350248+0000","last_peered":"2024-03-18T15:57:58.350248+0000","last_clean":"2024-03-18T15:57:58.350248+0000","last_became_active":"2024-03-18T15:57:57.181837+0000","last_became_peered":"2024-03-18T15:57:57.181837+0000","last_unstale":"2024-03-18T15:57:58.350248+0000","last_undegraded":"2024-03-18T15:57:58.350248+0000","last_fullsized":"2024-03-18T15:57:58.350248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:37:32.225077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594440+0000","last_change":"2024-03-18T15:57:56.171691+0000","last_active":"2024-03-18T15:57:58.594440+0000","last_peered":"2024-03-18T15:57:58.594440+0000","last_clean":"2024-03-18T15:57:58.594440+0000","last_became_active":"2024-03-18T15:57:56.171241+0000","last_became_peered":"2024-03-18T15:57:56.171241+0000","last_unstale":"2024-03-18T15:57:58.594440+0000","last_undegraded":"2024-03-18T15:57:58.594440+0000","last_fullsized":"2024-03-18T15:57:58.594440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:50.411828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173550+0000","last_change":"2024-03-18T15:57:57.191757+0000","last_active":"2024-03-18T15:57:58.173550+0000","last_peered":"2024-03-18T15:57:58.173550+0000","last_clean":"2024-03-18T15:57:58.173550+0000","last_became_active":"2024-03-18T15:57:57.191474+0000","last_became_peered":"2024-03-18T15:57:57.191474+0000","last_unstale":"2024-03-18T15:57:58.173550+0000","last_undegraded":"2024-03-18T15:57:58.173550+0000","last_fullsized":"2024-03-18T15:57:58.173550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:18.430632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173709+0000","last_change":"2024-03-18T15:57:56.176784+0000","last_active":"2024-03-18T15:57:58.173709+0000","last_peered":"2024-03-18T15:57:58.173709+0000","last_clean":"2024-03-18T15:57:58.173709+0000","last_became_active":"2024-03-18T15:57:56.176468+0000","last_became_peered":"2024-03-18T15:57:56.176468+0000","last_unstale":"2024-03-18T15:57:58.173709+0000","last_undegraded":"2024-03-18T15:57:58.173709+0000","last_fullsized":"2024-03-18T15:57:58.173709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:50.941724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170451+0000","last_change":"2024-03-18T15:57:57.183492+0000","last_active":"2024-03-18T15:57:59.170451+0000","last_peered":"2024-03-18T15:57:59.170451+0000","last_clean":"2024-03-18T15:57:59.170451+0000","last_became_active":"2024-03-18T15:57:57.183156+0000","last_became_peered":"2024-03-18T15:57:57.183156+0000","last_unstale":"2024-03-18T15:57:59.170451+0000","last_undegraded":"2024-03-18T15:57:59.170451+0000","last_fullsized":"2024-03-18T15:57:59.170451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:41.441788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167791+0000","last_change":"2024-03-18T15:57:56.176614+0000","last_active":"2024-03-18T15:57:59.167791+0000","last_peered":"2024-03-18T15:57:59.167791+0000","last_clean":"2024-03-18T15:57:59.167791+0000","last_became_active":"2024-03-18T15:57:56.176245+0000","last_became_peered":"2024-03-18T15:57:56.176245+0000","last_unstale":"2024-03-18T15:57:59.167791+0000","last_undegraded":"2024-03-18T15:57:59.167791+0000","last_fullsized":"2024-03-18T15:57:59.167791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:11.647223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173783+0000","last_change":"2024-03-18T15:57:57.180588+0000","last_active":"2024-03-18T15:57:58.173783+0000","last_peered":"2024-03-18T15:57:58.173783+0000","last_clean":"2024-03-18T15:57:58.173783+0000","last_became_active":"2024-03-18T15:57:57.180245+0000","last_became_peered":"2024-03-18T15:57:57.180245+0000","last_unstale":"2024-03-18T15:57:58.173783+0000","last_undegraded":"2024-03-18T15:57:58.173783+0000","last_fullsized":"2024-03-18T15:57:58.173783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:38.095406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167593+0000","last_change":"2024-03-18T15:57:56.179815+0000","last_active":"2024-03-18T15:57:59.167593+0000","last_peered":"2024-03-18T15:57:59.167593+0000","last_clean":"2024-03-18T15:57:59.167593+0000","last_became_active":"2024-03-18T15:57:56.179495+0000","last_became_peered":"2024-03-18T15:57:56.179495+0000","last_unstale":"2024-03-18T15:57:59.167593+0000","last_undegraded":"2024-03-18T15:57:59.167593+0000","last_fullsized":"2024-03-18T15:57:59.167593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:17.876766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174164+0000","last_change":"2024-03-18T15:57:57.180021+0000","last_active":"2024-03-18T15:57:58.174164+0000","last_peered":"2024-03-18T15:57:58.174164+0000","last_clean":"2024-03-18T15:57:58.174164+0000","last_became_active":"2024-03-18T15:57:57.179642+0000","last_became_peered":"2024-03-18T15:57:57.179642+0000","last_unstale":"2024-03-18T15:57:58.174164+0000","last_undegraded":"2024-03-18T15:57:58.174164+0000","last_fullsized":"2024-03-18T15:57:58.174164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:36.573584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:01.202331+0000","last_change":"2024-03-18T15:57:56.178907+0000","last_active":"2024-03-18T15:58:01.202331+0000","last_peered":"2024-03-18T15:58:01.202331+0000","last_clean":"2024-03-18T15:58:01.202331+0000","last_became_active":"2024-03-18T15:57:56.178420+0000","last_became_peered":"2024-03-18T15:57:56.178420+0000","last_unstale":"2024-03-18T15:58:01.202331+0000","last_undegraded":"2024-03-18T15:58:01.202331+0000","last_fullsized":"2024-03-18T15:58:01.202331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:24:45.513178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167581+0000","last_change":"2024-03-18T15:57:57.187726+0000","last_active":"2024-03-18T15:57:59.167581+0000","last_peered":"2024-03-18T15:57:59.167581+0000","last_clean":"2024-03-18T15:57:59.167581+0000","last_became_active":"2024-03-18T15:57:57.187383+0000","last_became_peered":"2024-03-18T15:57:57.187383+0000","last_unstale":"2024-03-18T15:57:59.167581+0000","last_undegraded":"2024-03-18T15:57:59.167581+0000","last_fullsized":"2024-03-18T15:57:59.167581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:23.134839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171337+0000","last_change":"2024-03-18T15:57:56.175120+0000","last_active":"2024-03-18T15:57:59.171337+0000","last_peered":"2024-03-18T15:57:59.171337+0000","last_clean":"2024-03-18T15:57:59.171337+0000","last_became_active":"2024-03-18T15:57:56.174554+0000","last_became_peered":"2024-03-18T15:57:56.174554+0000","last_unstale":"2024-03-18T15:57:59.171337+0000","last_undegraded":"2024-03-18T15:57:59.171337+0000","last_fullsized":"2024-03-18T15:57:59.171337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:16.894313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174247+0000","last_change":"2024-03-18T15:57:57.192612+0000","last_active":"2024-03-18T15:57:58.174247+0000","last_peered":"2024-03-18T15:57:58.174247+0000","last_clean":"2024-03-18T15:57:58.174247+0000","last_became_active":"2024-03-18T15:57:57.192284+0000","last_became_peered":"2024-03-18T15:57:57.192284+0000","last_unstale":"2024-03-18T15:57:58.174247+0000","last_undegraded":"2024-03-18T15:57:58.174247+0000","last_fullsized":"2024-03-18T15:57:58.174247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:53.597362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198580+0000","last_change":"2024-03-18T15:57:56.165164+0000","last_active":"2024-03-18T15:58:00.198580+0000","last_peered":"2024-03-18T15:58:00.198580+0000","last_clean":"2024-03-18T15:58:00.198580+0000","last_became_active":"2024-03-18T15:57:56.164844+0000","last_became_peered":"2024-03-18T15:57:56.164844+0000","last_unstale":"2024-03-18T15:58:00.198580+0000","last_undegraded":"2024-03-18T15:58:00.198580+0000","last_fullsized":"2024-03-18T15:58:00.198580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:36.754823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173060+0000","last_change":"2024-03-18T15:57:57.180176+0000","last_active":"2024-03-18T15:57:58.173060+0000","last_peered":"2024-03-18T15:57:58.173060+0000","last_clean":"2024-03-18T15:57:58.173060+0000","last_became_active":"2024-03-18T15:57:57.179832+0000","last_became_peered":"2024-03-18T15:57:57.179832+0000","last_unstale":"2024-03-18T15:57:58.173060+0000","last_undegraded":"2024-03-18T15:57:58.173060+0000","last_fullsized":"2024-03-18T15:57:58.173060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:32.249126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173739+0000","last_change":"2024-03-18T15:57:56.175404+0000","last_active":"2024-03-18T15:57:58.173739+0000","last_peered":"2024-03-18T15:57:58.173739+0000","last_clean":"2024-03-18T15:57:58.173739+0000","last_became_active":"2024-03-18T15:57:56.174877+0000","last_became_peered":"2024-03-18T15:57:56.174877+0000","last_unstale":"2024-03-18T15:57:58.173739+0000","last_undegraded":"2024-03-18T15:57:58.173739+0000","last_fullsized":"2024-03-18T15:57:58.173739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:39.536240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594506+0000","last_change":"2024-03-18T15:57:57.178213+0000","last_active":"2024-03-18T15:57:58.594506+0000","last_peered":"2024-03-18T15:57:58.594506+0000","last_clean":"2024-03-18T15:57:58.594506+0000","last_became_active":"2024-03-18T15:57:57.177863+0000","last_became_peered":"2024-03-18T15:57:57.177863+0000","last_unstale":"2024-03-18T15:57:58.594506+0000","last_undegraded":"2024-03-18T15:57:58.594506+0000","last_fullsized":"2024-03-18T15:57:58.594506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:42.988890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190571+0000","last_change":"2024-03-18T15:57:56.172506+0000","last_active":"2024-03-18T15:57:59.190571+0000","last_peered":"2024-03-18T15:57:59.190571+0000","last_clean":"2024-03-18T15:57:59.190571+0000","last_became_active":"2024-03-18T15:57:56.172153+0000","last_became_peered":"2024-03-18T15:57:56.172153+0000","last_unstale":"2024-03-18T15:57:59.190571+0000","last_undegraded":"2024-03-18T15:57:59.190571+0000","last_fullsized":"2024-03-18T15:57:59.190571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:30.563167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173965+0000","last_change":"2024-03-18T15:57:57.181506+0000","last_active":"2024-03-18T15:57:58.173965+0000","last_peered":"2024-03-18T15:57:58.173965+0000","last_clean":"2024-03-18T15:57:58.173965+0000","last_became_active":"2024-03-18T15:57:57.181139+0000","last_became_peered":"2024-03-18T15:57:57.181139+0000","last_unstale":"2024-03-18T15:57:58.173965+0000","last_undegraded":"2024-03-18T15:57:58.173965+0000","last_fullsized":"2024-03-18T15:57:58.173965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:40.239719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188980+0000","last_change":"2024-03-18T15:57:56.170895+0000","last_active":"2024-03-18T15:57:59.188980+0000","last_peered":"2024-03-18T15:57:59.188980+0000","last_clean":"2024-03-18T15:57:59.188980+0000","last_became_active":"2024-03-18T15:57:56.170394+0000","last_became_peered":"2024-03-18T15:57:56.170394+0000","last_unstale":"2024-03-18T15:57:59.188980+0000","last_undegraded":"2024-03-18T15:57:59.188980+0000","last_fullsized":"2024-03-18T15:57:59.188980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.915841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170192+0000","last_change":"2024-03-18T15:57:57.186766+0000","last_active":"2024-03-18T15:57:59.170192+0000","last_peered":"2024-03-18T15:57:59.170192+0000","last_clean":"2024-03-18T15:57:59.170192+0000","last_became_active":"2024-03-18T15:57:57.186453+0000","last_became_peered":"2024-03-18T15:57:57.186453+0000","last_unstale":"2024-03-18T15:57:59.170192+0000","last_undegraded":"2024-03-18T15:57:59.170192+0000","last_fullsized":"2024-03-18T15:57:59.170192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:19.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188521+0000","last_change":"2024-03-18T15:57:56.174536+0000","last_active":"2024-03-18T15:57:59.188521+0000","last_peered":"2024-03-18T15:57:59.188521+0000","last_clean":"2024-03-18T15:57:59.188521+0000","last_became_active":"2024-03-18T15:57:56.174216+0000","last_became_peered":"2024-03-18T15:57:56.174216+0000","last_unstale":"2024-03-18T15:57:59.188521+0000","last_undegraded":"2024-03-18T15:57:59.188521+0000","last_fullsized":"2024-03-18T15:57:59.188521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:50.324174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173861+0000","last_change":"2024-03-18T15:57:57.178511+0000","last_active":"2024-03-18T15:57:58.173861+0000","last_peered":"2024-03-18T15:57:58.173861+0000","last_clean":"2024-03-18T15:57:58.173861+0000","last_became_active":"2024-03-18T15:57:57.178117+0000","last_became_peered":"2024-03-18T15:57:57.178117+0000","last_unstale":"2024-03-18T15:57:58.173861+0000","last_undegraded":"2024-03-18T15:57:58.173861+0000","last_fullsized":"2024-03-18T15:57:58.173861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:52.891086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.189683+0000","last_change":"2024-03-18T15:57:56.173372+0000","last_active":"2024-03-18T15:57:59.189683+0000","last_peered":"2024-03-18T15:57:59.189683+0000","last_clean":"2024-03-18T15:57:59.189683+0000","last_became_active":"2024-03-18T15:57:56.173049+0000","last_became_peered":"2024-03-18T15:57:56.173049+0000","last_unstale":"2024-03-18T15:57:59.189683+0000","last_undegraded":"2024-03-18T15:57:59.189683+0000","last_fullsized":"2024-03-18T15:57:59.189683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:01.655747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167646+0000","last_change":"2024-03-18T15:57:57.183725+0000","last_active":"2024-03-18T15:57:59.167646+0000","last_peered":"2024-03-18T15:57:59.167646+0000","last_clean":"2024-03-18T15:57:59.167646+0000","last_became_active":"2024-03-18T15:57:57.183391+0000","last_became_peered":"2024-03-18T15:57:57.183391+0000","last_unstale":"2024-03-18T15:57:59.167646+0000","last_undegraded":"2024-03-18T15:57:59.167646+0000","last_fullsized":"2024-03-18T15:57:59.167646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:42.945519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198950+0000","last_change":"2024-03-18T15:57:56.179534+0000","last_active":"2024-03-18T15:58:00.198950+0000","last_peered":"2024-03-18T15:58:00.198950+0000","last_clean":"2024-03-18T15:58:00.198950+0000","last_became_active":"2024-03-18T15:57:56.179217+0000","last_became_peered":"2024-03-18T15:57:56.179217+0000","last_unstale":"2024-03-18T15:58:00.198950+0000","last_undegraded":"2024-03-18T15:58:00.198950+0000","last_fullsized":"2024-03-18T15:58:00.198950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:19.176423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169660+0000","last_change":"2024-03-18T15:57:57.188958+0000","last_active":"2024-03-18T15:57:59.169660+0000","last_peered":"2024-03-18T15:57:59.169660+0000","last_clean":"2024-03-18T15:57:59.169660+0000","last_became_active":"2024-03-18T15:57:57.188586+0000","last_became_peered":"2024-03-18T15:57:57.188586+0000","last_unstale":"2024-03-18T15:57:59.169660+0000","last_undegraded":"2024-03-18T15:57:59.169660+0000","last_fullsized":"2024-03-18T15:57:59.169660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:48.122539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188281+0000","last_change":"2024-03-18T15:57:56.180530+0000","last_active":"2024-03-18T15:57:59.188281+0000","last_peered":"2024-03-18T15:57:59.188281+0000","last_clean":"2024-03-18T15:57:59.188281+0000","last_became_active":"2024-03-18T15:57:56.180019+0000","last_became_peered":"2024-03-18T15:57:56.180019+0000","last_unstale":"2024-03-18T15:57:59.188281+0000","last_undegraded":"2024-03-18T15:57:59.188281+0000","last_fullsized":"2024-03-18T15:57:59.188281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:04.119730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171093+0000","last_change":"2024-03-18T15:57:57.175201+0000","last_active":"2024-03-18T15:57:59.171093+0000","last_peered":"2024-03-18T15:57:59.171093+0000","last_clean":"2024-03-18T15:57:59.171093+0000","last_became_active":"2024-03-18T15:57:57.174878+0000","last_became_peered":"2024-03-18T15:57:57.174878+0000","last_unstale":"2024-03-18T15:57:59.171093+0000","last_undegraded":"2024-03-18T15:57:59.171093+0000","last_fullsized":"2024-03-18T15:57:59.171093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:43.545594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198179+0000","last_change":"2024-03-18T15:57:56.172587+0000","last_active":"2024-03-18T15:58:00.198179+0000","last_peered":"2024-03-18T15:58:00.198179+0000","last_clean":"2024-03-18T15:58:00.198179+0000","last_became_active":"2024-03-18T15:57:56.172246+0000","last_became_peered":"2024-03-18T15:57:56.172246+0000","last_unstale":"2024-03-18T15:58:00.198179+0000","last_undegraded":"2024-03-18T15:58:00.198179+0000","last_fullsized":"2024-03-18T15:58:00.198179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.543059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169907+0000","last_change":"2024-03-18T15:57:57.181899+0000","last_active":"2024-03-18T15:57:59.169907+0000","last_peered":"2024-03-18T15:57:59.169907+0000","last_clean":"2024-03-18T15:57:59.169907+0000","last_became_active":"2024-03-18T15:57:57.181559+0000","last_became_peered":"2024-03-18T15:57:57.181559+0000","last_unstale":"2024-03-18T15:57:59.169907+0000","last_undegraded":"2024-03-18T15:57:59.169907+0000","last_fullsized":"2024-03-18T15:57:59.169907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:17.504153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190849+0000","last_change":"2024-03-18T15:57:56.179375+0000","last_active":"2024-03-18T15:57:59.190849+0000","last_peered":"2024-03-18T15:57:59.190849+0000","last_clean":"2024-03-18T15:57:59.190849+0000","last_became_active":"2024-03-18T15:57:56.178990+0000","last_became_peered":"2024-03-18T15:57:56.178990+0000","last_unstale":"2024-03-18T15:57:59.190849+0000","last_undegraded":"2024-03-18T15:57:59.190849+0000","last_fullsized":"2024-03-18T15:57:59.190849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:36.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173781+0000","last_change":"2024-03-18T15:57:57.188135+0000","last_active":"2024-03-18T15:57:58.173781+0000","last_peered":"2024-03-18T15:57:58.173781+0000","last_clean":"2024-03-18T15:57:58.173781+0000","last_became_active":"2024-03-18T15:57:57.187811+0000","last_became_peered":"2024-03-18T15:57:57.187811+0000","last_unstale":"2024-03-18T15:57:58.173781+0000","last_undegraded":"2024-03-18T15:57:58.173781+0000","last_fullsized":"2024-03-18T15:57:58.173781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:11.235420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167711+0000","last_change":"2024-03-18T15:57:56.181774+0000","last_active":"2024-03-18T15:57:59.167711+0000","last_peered":"2024-03-18T15:57:59.167711+0000","last_clean":"2024-03-18T15:57:59.167711+0000","last_became_active":"2024-03-18T15:57:56.181360+0000","last_became_peered":"2024-03-18T15:57:56.181360+0000","last_unstale":"2024-03-18T15:57:59.167711+0000","last_undegraded":"2024-03-18T15:57:59.167711+0000","last_fullsized":"2024-03-18T15:57:59.167711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:08.367151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169680+0000","last_change":"2024-03-18T15:57:57.188960+0000","last_active":"2024-03-18T15:57:59.169680+0000","last_peered":"2024-03-18T15:57:59.169680+0000","last_clean":"2024-03-18T15:57:59.169680+0000","last_became_active":"2024-03-18T15:57:57.187528+0000","last_became_peered":"2024-03-18T15:57:57.187528+0000","last_unstale":"2024-03-18T15:57:59.169680+0000","last_undegraded":"2024-03-18T15:57:59.169680+0000","last_fullsized":"2024-03-18T15:57:59.169680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:29.259977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594450+0000","last_change":"2024-03-18T15:57:56.174055+0000","last_active":"2024-03-18T15:57:58.594450+0000","last_peered":"2024-03-18T15:57:58.594450+0000","last_clean":"2024-03-18T15:57:58.594450+0000","last_became_active":"2024-03-18T15:57:56.173736+0000","last_became_peered":"2024-03-18T15:57:56.173736+0000","last_unstale":"2024-03-18T15:57:58.594450+0000","last_undegraded":"2024-03-18T15:57:58.594450+0000","last_fullsized":"2024-03-18T15:57:58.594450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:00.319049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170834+0000","last_change":"2024-03-18T15:57:57.189238+0000","last_active":"2024-03-18T15:57:59.170834+0000","last_peered":"2024-03-18T15:57:59.170834+0000","last_clean":"2024-03-18T15:57:59.170834+0000","last_became_active":"2024-03-18T15:57:57.188900+0000","last_became_peered":"2024-03-18T15:57:57.188900+0000","last_unstale":"2024-03-18T15:57:59.170834+0000","last_undegraded":"2024-03-18T15:57:59.170834+0000","last_fullsized":"2024-03-18T15:57:59.170834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:16.810976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170265+0000","last_change":"2024-03-18T15:57:56.178765+0000","last_active":"2024-03-18T15:57:59.170265+0000","last_peered":"2024-03-18T15:57:59.170265+0000","last_clean":"2024-03-18T15:57:59.170265+0000","last_became_active":"2024-03-18T15:57:56.178399+0000","last_became_peered":"2024-03-18T15:57:56.178399+0000","last_unstale":"2024-03-18T15:57:59.170265+0000","last_undegraded":"2024-03-18T15:57:59.170265+0000","last_fullsized":"2024-03-18T15:57:59.170265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:46:07.844453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171300+0000","last_change":"2024-03-18T15:57:57.173968+0000","last_active":"2024-03-18T15:57:59.171300+0000","last_peered":"2024-03-18T15:57:59.171300+0000","last_clean":"2024-03-18T15:57:59.171300+0000","last_became_active":"2024-03-18T15:57:57.173605+0000","last_became_peered":"2024-03-18T15:57:57.173605+0000","last_unstale":"2024-03-18T15:57:59.171300+0000","last_undegraded":"2024-03-18T15:57:59.171300+0000","last_fullsized":"2024-03-18T15:57:59.171300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:34.676393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174385+0000","last_change":"2024-03-18T15:57:57.186561+0000","last_active":"2024-03-18T15:57:58.174385+0000","last_peered":"2024-03-18T15:57:58.174385+0000","last_clean":"2024-03-18T15:57:58.174385+0000","last_became_active":"2024-03-18T15:57:57.186260+0000","last_became_peered":"2024-03-18T15:57:57.186260+0000","last_unstale":"2024-03-18T15:57:58.174385+0000","last_undegraded":"2024-03-18T15:57:58.174385+0000","last_fullsized":"2024-03-18T15:57:58.174385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:54:06.646480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.200785+0000","last_change":"2024-03-18T15:57:56.164175+0000","last_active":"2024-03-18T15:58:00.200785+0000","last_peered":"2024-03-18T15:58:00.200785+0000","last_clean":"2024-03-18T15:58:00.200785+0000","last_became_active":"2024-03-18T15:57:56.163724+0000","last_became_peered":"2024-03-18T15:57:56.163724+0000","last_unstale":"2024-03-18T15:58:00.200785+0000","last_undegraded":"2024-03-18T15:58:00.200785+0000","last_fullsized":"2024-03-18T15:58:00.200785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:48:53.596211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173257+0000","last_change":"2024-03-18T15:57:57.191507+0000","last_active":"2024-03-18T15:57:58.173257+0000","last_peered":"2024-03-18T15:57:58.173257+0000","last_clean":"2024-03-18T15:57:58.173257+0000","last_became_active":"2024-03-18T15:57:57.191182+0000","last_became_peered":"2024-03-18T15:57:57.191182+0000","last_unstale":"2024-03-18T15:57:58.173257+0000","last_undegraded":"2024-03-18T15:57:58.173257+0000","last_fullsized":"2024-03-18T15:57:58.173257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:39.878675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188475+0000","last_change":"2024-03-18T15:57:56.169490+0000","last_active":"2024-03-18T15:57:59.188475+0000","last_peered":"2024-03-18T15:57:59.188475+0000","last_clean":"2024-03-18T15:57:59.188475+0000","last_became_active":"2024-03-18T15:57:56.169107+0000","last_became_peered":"2024-03-18T15:57:56.169107+0000","last_unstale":"2024-03-18T15:57:59.188475+0000","last_undegraded":"2024-03-18T15:57:59.188475+0000","last_fullsized":"2024-03-18T15:57:59.188475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:59.722909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169906+0000","last_change":"2024-03-18T15:56:43.270716+0000","last_active":"2024-03-18T15:57:59.169906+0000","last_peered":"2024-03-18T15:57:59.169906+0000","last_clean":"2024-03-18T15:57:59.169906+0000","last_became_active":"2024-03-18T15:56:43.270316+0000","last_became_peered":"2024-03-18T15:56:43.270316+0000","last_unstale":"2024-03-18T15:57:59.169906+0000","last_undegraded":"2024-03-18T15:57:59.169906+0000","last_fullsized":"2024-03-18T15:57:59.169906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_clean_scrub_stamp":"2024-03-18T15:53:17.916048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:00.779948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169444+0000","last_change":"2024-03-18T15:57:57.172265+0000","last_active":"2024-03-18T15:57:59.169444+0000","last_peered":"2024-03-18T15:57:59.169444+0000","last_clean":"2024-03-18T15:57:59.169444+0000","last_became_active":"2024-03-18T15:57:57.171962+0000","last_became_peered":"2024-03-18T15:57:57.171962+0000","last_unstale":"2024-03-18T15:57:59.169444+0000","last_undegraded":"2024-03-18T15:57:59.169444+0000","last_fullsized":"2024-03-18T15:57:59.169444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:41:44.402012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198203+0000","last_change":"2024-03-18T15:57:56.176528+0000","last_active":"2024-03-18T15:58:00.198203+0000","last_peered":"2024-03-18T15:58:00.198203+0000","last_clean":"2024-03-18T15:58:00.198203+0000","last_became_active":"2024-03-18T15:57:56.176137+0000","last_became_peered":"2024-03-18T15:57:56.176137+0000","last_unstale":"2024-03-18T15:58:00.198203+0000","last_undegraded":"2024-03-18T15:58:00.198203+0000","last_fullsized":"2024-03-18T15:58:00.198203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:19:14.299323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190946+0000","last_change":"2024-03-18T15:57:56.178710+0000","last_active":"2024-03-18T15:57:59.190946+0000","last_peered":"2024-03-18T15:57:59.190946+0000","last_clean":"2024-03-18T15:57:59.190946+0000","last_became_active":"2024-03-18T15:57:56.178388+0000","last_became_peered":"2024-03-18T15:57:56.178388+0000","last_unstale":"2024-03-18T15:57:59.190946+0000","last_undegraded":"2024-03-18T15:57:59.190946+0000","last_fullsized":"2024-03-18T15:57:59.190946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:23.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170074+0000","last_change":"2024-03-18T15:57:57.168951+0000","last_active":"2024-03-18T15:57:59.170074+0000","last_peered":"2024-03-18T15:57:59.170074+0000","last_clean":"2024-03-18T15:57:59.170074+0000","last_became_active":"2024-03-18T15:57:57.168567+0000","last_became_peered":"2024-03-18T15:57:57.168567+0000","last_unstale":"2024-03-18T15:57:59.170074+0000","last_undegraded":"2024-03-18T15:57:59.170074+0000","last_fullsized":"2024-03-18T15:57:59.170074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:41.171254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594622+0000","last_change":"2024-03-18T15:57:56.171665+0000","last_active":"2024-03-18T15:57:58.594622+0000","last_peered":"2024-03-18T15:57:58.594622+0000","last_clean":"2024-03-18T15:57:58.594622+0000","last_became_active":"2024-03-18T15:57:56.171282+0000","last_became_peered":"2024-03-18T15:57:56.171282+0000","last_unstale":"2024-03-18T15:57:58.594622+0000","last_undegraded":"2024-03-18T15:57:58.594622+0000","last_fullsized":"2024-03-18T15:57:58.594622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:03:27.523561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173508+0000","last_change":"2024-03-18T15:57:57.185647+0000","last_active":"2024-03-18T15:57:58.173508+0000","last_peered":"2024-03-18T15:57:58.173508+0000","last_clean":"2024-03-18T15:57:58.173508+0000","last_became_active":"2024-03-18T15:57:57.185352+0000","last_became_peered":"2024-03-18T15:57:57.185352+0000","last_unstale":"2024-03-18T15:57:58.173508+0000","last_undegraded":"2024-03-18T15:57:58.173508+0000","last_fullsized":"2024-03-18T15:57:58.173508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:38.247776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.186067+0000","last_change":"2024-03-18T15:57:56.170878+0000","last_active":"2024-03-18T15:57:59.186067+0000","last_peered":"2024-03-18T15:57:59.186067+0000","last_clean":"2024-03-18T15:57:59.186067+0000","last_became_active":"2024-03-18T15:57:56.170558+0000","last_became_peered":"2024-03-18T15:57:56.170558+0000","last_unstale":"2024-03-18T15:57:59.186067+0000","last_undegraded":"2024-03-18T15:57:59.186067+0000","last_fullsized":"2024-03-18T15:57:59.186067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:26:39.963384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170622+0000","last_change":"2024-03-18T15:57:57.181012+0000","last_active":"2024-03-18T15:57:59.170622+0000","last_peered":"2024-03-18T15:57:59.170622+0000","last_clean":"2024-03-18T15:57:59.170622+0000","last_became_active":"2024-03-18T15:57:57.180613+0000","last_became_peered":"2024-03-18T15:57:57.180613+0000","last_unstale":"2024-03-18T15:57:59.170622+0000","last_undegraded":"2024-03-18T15:57:59.170622+0000","last_fullsized":"2024-03-18T15:57:59.170622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:22.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190523+0000","last_change":"2024-03-18T15:57:56.171929+0000","last_active":"2024-03-18T15:57:59.190523+0000","last_peered":"2024-03-18T15:57:59.190523+0000","last_clean":"2024-03-18T15:57:59.190523+0000","last_became_active":"2024-03-18T15:57:56.171570+0000","last_became_peered":"2024-03-18T15:57:56.171570+0000","last_unstale":"2024-03-18T15:57:59.190523+0000","last_undegraded":"2024-03-18T15:57:59.190523+0000","last_fullsized":"2024-03-18T15:57:59.190523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:11.579956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167389+0000","last_change":"2024-03-18T15:57:57.178118+0000","last_active":"2024-03-18T15:57:59.167389+0000","last_peered":"2024-03-18T15:57:59.167389+0000","last_clean":"2024-03-18T15:57:59.167389+0000","last_became_active":"2024-03-18T15:57:57.177764+0000","last_became_peered":"2024-03-18T15:57:57.177764+0000","last_unstale":"2024-03-18T15:57:59.167389+0000","last_undegraded":"2024-03-18T15:57:59.167389+0000","last_fullsized":"2024-03-18T15:57:59.167389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:07.519544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190597+0000","last_change":"2024-03-18T15:57:56.168485+0000","last_active":"2024-03-18T15:57:59.190597+0000","last_peered":"2024-03-18T15:57:59.190597+0000","last_clean":"2024-03-18T15:57:59.190597+0000","last_became_active":"2024-03-18T15:57:56.168124+0000","last_became_peered":"2024-03-18T15:57:56.168124+0000","last_unstale":"2024-03-18T15:57:59.190597+0000","last_undegraded":"2024-03-18T15:57:59.190597+0000","last_fullsized":"2024-03-18T15:57:59.190597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:09.465975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169540+0000","last_change":"2024-03-18T15:57:57.184781+0000","last_active":"2024-03-18T15:57:59.169540+0000","last_peered":"2024-03-18T15:57:59.169540+0000","last_clean":"2024-03-18T15:57:59.169540+0000","last_became_active":"2024-03-18T15:57:57.184431+0000","last_became_peered":"2024-03-18T15:57:57.184431+0000","last_unstale":"2024-03-18T15:57:59.169540+0000","last_undegraded":"2024-03-18T15:57:59.169540+0000","last_fullsized":"2024-03-18T15:57:59.169540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:32.945288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350312+0000","last_change":"2024-03-18T15:57:56.180295+0000","last_active":"2024-03-18T15:57:58.350312+0000","last_peered":"2024-03-18T15:57:58.350312+0000","last_clean":"2024-03-18T15:57:58.350312+0000","last_became_active":"2024-03-18T15:57:56.179956+0000","last_became_peered":"2024-03-18T15:57:56.179956+0000","last_unstale":"2024-03-18T15:57:58.350312+0000","last_undegraded":"2024-03-18T15:57:58.350312+0000","last_fullsized":"2024-03-18T15:57:58.350312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:49.664982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173501+0000","last_change":"2024-03-18T15:57:57.187421+0000","last_active":"2024-03-18T15:57:58.173501+0000","last_peered":"2024-03-18T15:57:58.173501+0000","last_clean":"2024-03-18T15:57:58.173501+0000","last_became_active":"2024-03-18T15:57:57.185978+0000","last_became_peered":"2024-03-18T15:57:57.185978+0000","last_unstale":"2024-03-18T15:57:58.173501+0000","last_undegraded":"2024-03-18T15:57:58.173501+0000","last_fullsized":"2024-03-18T15:57:58.173501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:07:59.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190461+0000","last_change":"2024-03-18T15:57:56.178118+0000","last_active":"2024-03-18T15:57:59.190461+0000","last_peered":"2024-03-18T15:57:59.190461+0000","last_clean":"2024-03-18T15:57:59.190461+0000","last_became_active":"2024-03-18T15:57:56.177758+0000","last_became_peered":"2024-03-18T15:57:56.177758+0000","last_unstale":"2024-03-18T15:57:59.190461+0000","last_undegraded":"2024-03-18T15:57:59.190461+0000","last_fullsized":"2024-03-18T15:57:59.190461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:34.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170395+0000","last_change":"2024-03-18T15:57:57.182666+0000","last_active":"2024-03-18T15:57:59.170395+0000","last_peered":"2024-03-18T15:57:59.170395+0000","last_clean":"2024-03-18T15:57:59.170395+0000","last_became_active":"2024-03-18T15:57:57.182328+0000","last_became_peered":"2024-03-18T15:57:57.182328+0000","last_unstale":"2024-03-18T15:57:59.170395+0000","last_undegraded":"2024-03-18T15:57:59.170395+0000","last_fullsized":"2024-03-18T15:57:59.170395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:55.132441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.197362+0000","last_change":"2024-03-18T15:57:56.177101+0000","last_active":"2024-03-18T15:58:00.197362+0000","last_peered":"2024-03-18T15:58:00.197362+0000","last_clean":"2024-03-18T15:58:00.197362+0000","last_became_active":"2024-03-18T15:57:56.176734+0000","last_became_peered":"2024-03-18T15:57:56.176734+0000","last_unstale":"2024-03-18T15:58:00.197362+0000","last_undegraded":"2024-03-18T15:58:00.197362+0000","last_fullsized":"2024-03-18T15:58:00.197362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:55.406210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170968+0000","last_change":"2024-03-18T15:57:57.183892+0000","last_active":"2024-03-18T15:57:59.170968+0000","last_peered":"2024-03-18T15:57:59.170968+0000","last_clean":"2024-03-18T15:57:59.170968+0000","last_became_active":"2024-03-18T15:57:57.183121+0000","last_became_peered":"2024-03-18T15:57:57.183121+0000","last_unstale":"2024-03-18T15:57:59.170968+0000","last_undegraded":"2024-03-18T15:57:59.170968+0000","last_fullsized":"2024-03-18T15:57:59.170968+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:11:13.504169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174105+0000","last_change":"2024-03-18T15:57:56.172479+0000","last_active":"2024-03-18T15:57:58.174105+0000","last_peered":"2024-03-18T15:57:58.174105+0000","last_clean":"2024-03-18T15:57:58.174105+0000","last_became_active":"2024-03-18T15:57:56.172120+0000","last_became_peered":"2024-03-18T15:57:56.172120+0000","last_unstale":"2024-03-18T15:57:58.174105+0000","last_undegraded":"2024-03-18T15:57:58.174105+0000","last_fullsized":"2024-03-18T15:57:58.174105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:15.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170482+0000","last_change":"2024-03-18T15:57:57.183674+0000","last_active":"2024-03-18T15:57:59.170482+0000","last_peered":"2024-03-18T15:57:59.170482+0000","last_clean":"2024-03-18T15:57:59.170482+0000","last_became_active":"2024-03-18T15:57:57.183270+0000","last_became_peered":"2024-03-18T15:57:57.183270+0000","last_unstale":"2024-03-18T15:57:59.170482+0000","last_undegraded":"2024-03-18T15:57:59.170482+0000","last_fullsized":"2024-03-18T15:57:59.170482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:43.258995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169735+0000","last_change":"2024-03-18T15:57:56.174497+0000","last_active":"2024-03-18T15:57:59.169735+0000","last_peered":"2024-03-18T15:57:59.169735+0000","last_clean":"2024-03-18T15:57:59.169735+0000","last_became_active":"2024-03-18T15:57:56.173862+0000","last_became_peered":"2024-03-18T15:57:56.173862+0000","last_unstale":"2024-03-18T15:57:59.169735+0000","last_undegraded":"2024-03-18T15:57:59.169735+0000","last_fullsized":"2024-03-18T15:57:59.169735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:00.362276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171330+0000","last_change":"2024-03-18T15:57:57.179510+0000","last_active":"2024-03-18T15:57:59.171330+0000","last_peered":"2024-03-18T15:57:59.171330+0000","last_clean":"2024-03-18T15:57:59.171330+0000","last_became_active":"2024-03-18T15:57:57.179178+0000","last_became_peered":"2024-03-18T15:57:57.179178+0000","last_unstale":"2024-03-18T15:57:59.171330+0000","last_undegraded":"2024-03-18T15:57:59.171330+0000","last_fullsized":"2024-03-18T15:57:59.171330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:46.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169602+0000","last_change":"2024-03-18T15:57:57.186842+0000","last_active":"2024-03-18T15:57:59.169602+0000","last_peered":"2024-03-18T15:57:59.169602+0000","last_clean":"2024-03-18T15:57:59.169602+0000","last_became_active":"2024-03-18T15:57:57.186501+0000","last_became_peered":"2024-03-18T15:57:57.186501+0000","last_unstale":"2024-03-18T15:57:59.169602+0000","last_undegraded":"2024-03-18T15:57:59.169602+0000","last_fullsized":"2024-03-18T15:57:59.169602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:11.988494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350322+0000","last_change":"2024-03-18T15:57:57.177630+0000","last_active":"2024-03-18T15:57:58.350322+0000","last_peered":"2024-03-18T15:57:58.350322+0000","last_clean":"2024-03-18T15:57:58.350322+0000","last_became_active":"2024-03-18T15:57:57.177227+0000","last_became_peered":"2024-03-18T15:57:57.177227+0000","last_unstale":"2024-03-18T15:57:58.350322+0000","last_undegraded":"2024-03-18T15:57:58.350322+0000","last_fullsized":"2024-03-18T15:57:58.350322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:27.004129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350149+0000","last_change":"2024-03-18T15:57:57.189048+0000","last_active":"2024-03-18T15:57:58.350149+0000","last_peered":"2024-03-18T15:57:58.350149+0000","last_clean":"2024-03-18T15:57:58.350149+0000","last_became_active":"2024-03-18T15:57:57.188704+0000","last_became_peered":"2024-03-18T15:57:57.188704+0000","last_unstale":"2024-03-18T15:57:58.350149+0000","last_undegraded":"2024-03-18T15:57:58.350149+0000","last_fullsized":"2024-03-18T15:57:58.350149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:10:53.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170546+0000","last_change":"2024-03-18T15:57:57.178611+0000","last_active":"2024-03-18T15:57:59.170546+0000","last_peered":"2024-03-18T15:57:59.170546+0000","last_clean":"2024-03-18T15:57:59.170546+0000","last_became_active":"2024-03-18T15:57:57.178305+0000","last_became_peered":"2024-03-18T15:57:57.178305+0000","last_unstale":"2024-03-18T15:57:59.170546+0000","last_undegraded":"2024-03-18T15:57:59.170546+0000","last_fullsized":"2024-03-18T15:57:59.170546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:51.925278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169899+0000","last_change":"2024-03-18T15:57:57.177928+0000","last_active":"2024-03-18T15:57:59.169899+0000","last_peered":"2024-03-18T15:57:59.169899+0000","last_clean":"2024-03-18T15:57:59.169899+0000","last_became_active":"2024-03-18T15:57:57.177364+0000","last_became_peered":"2024-03-18T15:57:57.177364+0000","last_unstale":"2024-03-18T15:57:59.169899+0000","last_undegraded":"2024-03-18T15:57:59.169899+0000","last_fullsized":"2024-03-18T15:57:59.169899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:59.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169705+0000","last_change":"2024-03-18T15:57:57.188080+0000","last_active":"2024-03-18T15:57:59.169705+0000","last_peered":"2024-03-18T15:57:59.169705+0000","last_clean":"2024-03-18T15:57:59.169705+0000","last_became_active":"2024-03-18T15:57:57.187767+0000","last_became_peered":"2024-03-18T15:57:57.187767+0000","last_unstale":"2024-03-18T15:57:59.169705+0000","last_undegraded":"2024-03-18T15:57:59.169705+0000","last_fullsized":"2024-03-18T15:57:59.169705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:26.290262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350443+0000","last_change":"2024-03-18T15:57:57.178581+0000","last_active":"2024-03-18T15:57:58.350443+0000","last_peered":"2024-03-18T15:57:58.350443+0000","last_clean":"2024-03-18T15:57:58.350443+0000","last_became_active":"2024-03-18T15:57:57.178168+0000","last_became_peered":"2024-03-18T15:57:57.178168+0000","last_unstale":"2024-03-18T15:57:58.350443+0000","last_undegraded":"2024-03-18T15:57:58.350443+0000","last_fullsized":"2024-03-18T15:57:58.350443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:07:07.601959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173475+0000","last_change":"2024-03-18T15:57:57.188709+0000","last_active":"2024-03-18T15:57:58.173475+0000","last_peered":"2024-03-18T15:57:58.173475+0000","last_clean":"2024-03-18T15:57:58.173475+0000","last_became_active":"2024-03-18T15:57:57.187884+0000","last_became_peered":"2024-03-18T15:57:57.187884+0000","last_unstale":"2024-03-18T15:57:58.173475+0000","last_undegraded":"2024-03-18T15:57:58.173475+0000","last_fullsized":"2024-03-18T15:57:58.173475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:19.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173835+0000","last_change":"2024-03-18T15:57:57.182489+0000","last_active":"2024-03-18T15:57:58.173835+0000","last_peered":"2024-03-18T15:57:58.173835+0000","last_clean":"2024-03-18T15:57:58.173835+0000","last_became_active":"2024-03-18T15:57:57.176462+0000","last_became_peered":"2024-03-18T15:57:57.176462+0000","last_unstale":"2024-03-18T15:57:58.173835+0000","last_undegraded":"2024-03-18T15:57:58.173835+0000","last_fullsized":"2024-03-18T15:57:58.173835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:46.138422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173007+0000","last_change":"2024-03-18T15:57:57.185848+0000","last_active":"2024-03-18T15:57:58.173007+0000","last_peered":"2024-03-18T15:57:58.173007+0000","last_clean":"2024-03-18T15:57:58.173007+0000","last_became_active":"2024-03-18T15:57:57.185507+0000","last_became_peered":"2024-03-18T15:57:57.185507+0000","last_unstale":"2024-03-18T15:57:58.173007+0000","last_undegraded":"2024-03-18T15:57:58.173007+0000","last_fullsized":"2024-03-18T15:57:58.173007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:09.319325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173733+0000","last_change":"2024-03-18T15:57:57.189222+0000","last_active":"2024-03-18T15:57:58.173733+0000","last_peered":"2024-03-18T15:57:58.173733+0000","last_clean":"2024-03-18T15:57:58.173733+0000","last_became_active":"2024-03-18T15:57:57.188847+0000","last_became_peered":"2024-03-18T15:57:57.188847+0000","last_unstale":"2024-03-18T15:57:58.173733+0000","last_undegraded":"2024-03-18T15:57:58.173733+0000","last_fullsized":"2024-03-18T15:57:58.173733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:15.935552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169947+0000","last_change":"2024-03-18T15:57:57.183269+0000","last_active":"2024-03-18T15:57:59.169947+0000","last_peered":"2024-03-18T15:57:59.169947+0000","last_clean":"2024-03-18T15:57:59.169947+0000","last_became_active":"2024-03-18T15:57:57.182866+0000","last_became_peered":"2024-03-18T15:57:57.182866+0000","last_unstale":"2024-03-18T15:57:59.169947+0000","last_undegraded":"2024-03-18T15:57:59.169947+0000","last_fullsized":"2024-03-18T15:57:59.169947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:15:08.500842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169644+0000","last_change":"2024-03-18T15:57:57.179926+0000","last_active":"2024-03-18T15:57:59.169644+0000","last_peered":"2024-03-18T15:57:59.169644+0000","last_clean":"2024-03-18T15:57:59.169644+0000","last_became_active":"2024-03-18T15:57:57.179534+0000","last_became_peered":"2024-03-18T15:57:57.179534+0000","last_unstale":"2024-03-18T15:57:59.169644+0000","last_undegraded":"2024-03-18T15:57:59.169644+0000","last_fullsized":"2024-03-18T15:57:59.169644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:41.467292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285179,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5110,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":38796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":443597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":1026619,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":443799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":921600,"data_stored":435049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":1035175,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":443105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":445977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":443299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":443659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":1034859,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":26860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:58:03.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 -- 172.21.15.150:0/1993680264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 msgr2=0x7fa774108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 --2- 172.21.15.150:0/1993680264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa774108b40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009ca0 tx=0x7fa76c02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 -- 172.21.15.150:0/1993680264 shutdown_connections 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 --2- 172.21.15.150:0/1993680264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa774108b40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 --2- 172.21.15.150:0/1993680264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7741018f0 0x7fa774101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 --2- 172.21.15.150:0/1993680264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa774100700 0x7fa774100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 -- 172.21.15.150:0/1993680264 >> 172.21.15.150:0/1993680264 conn(0x7fa7740fbc70 msgr2=0x7fa7740fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 -- 172.21.15.150:0/1993680264 shutdown_connections 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.039+0000 7fa77904b640 1 -- 172.21.15.150:0/1993680264 wait complete. 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 Processor -- start 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 -- start start 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa774100700 0x7fa774197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7741018f0 0x7fa774197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c360 con 0x7fa7741022b0 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c1e0 con 0x7fa774100700 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77904b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa77410c4e0 con 0x7fa7741018f0 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:04.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55048/0 (socket says 172.21.15.150:55048) 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.040+0000 7fa77359e640 1 -- 172.21.15.150:0/2076150555 learned_addr learned my addr 172.21.15.150:0/2076150555 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77259c640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7741018f0 0x7fa774197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa772d9d640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa774100700 0x7fa774197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 -- 172.21.15.150:0/2076150555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7741018f0 msgr2=0x7fa774197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7741018f0 0x7fa774197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 -- 172.21.15.150:0/2076150555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa774100700 msgr2=0x7fa774197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa774100700 0x7fa774197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 -- 172.21.15.150:0/2076150555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77419f6d0 con 0x7fa7741022b0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77359e640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa76c037040 tx=0x7fa76c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c041020 con 0x7fa7741022b0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa76c034cd0 con 0x7fa7741022b0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77419f960 con 0x7fa7741022b0 2024-03-18T15:58:04.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c039040 con 0x7fa7741022b0 2024-03-18T15:58:04.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.041+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77419fe50 con 0x7fa7741022b0 2024-03-18T15:58:04.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.042+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7741a0020 con 0x7fa7741022b0 2024-03-18T15:58:04.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.043+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fa76c044020 con 0x7fa7741022b0 2024-03-18T15:58:04.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.044+0000 7fa75bfff640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 0x7fa75c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:04.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.044+0000 7fa772d9d640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 0x7fa75c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:04.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.044+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa76c0f07d0 con 0x7fa7741022b0 2024-03-18T15:58:04.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.044+0000 7fa772d9d640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 0x7fa75c0a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa774101750 tx=0x7fa768005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.048+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa76c04b020 con 0x7fa7741022b0 2024-03-18T15:58:04.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:04 smithi150 ceph-mon[31946]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:04.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:04 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/779490158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:58:04.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:04 smithi150 ceph-mon[31946]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 983 B/s rd, 4.1 KiB/s wr, 15 op/s 2024-03-18T15:58:04.214 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:04 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2329708125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:04.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.227+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa774069050 con 0x7fa75c0a2890 2024-03-18T15:58:04.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.234+0000 7fa75bfff640 1 -- 172.21.15.150:0/2076150555 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258916 (secure 0 0 0) 0x7fa774069050 con 0x7fa75c0a2890 2024-03-18T15:58:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:04.239 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 msgr2=0x7fa75c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 0x7fa75c0a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa774101750 tx=0x7fa768005f90 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 msgr2=0x7fa77419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa76c037040 tx=0x7fa76c032f40 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 shutdown_connections 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa75c0a2890 0x7fa75c0a4d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7741022b0 0x7fa77419f060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7741018f0 0x7fa774197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 --2- 172.21.15.150:0/2076150555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa774100700 0x7fa774197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:04.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 >> 172.21.15.150:0/2076150555 conn(0x7fa7740fbc70 msgr2=0x7fa774068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:04.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 shutdown_connections 2024-03-18T15:58:04.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:04.237+0000 7fa77904b640 1 -- 172.21.15.150:0/2076150555 wait complete. 2024-03-18T15:58:04.249 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-18T15:58:03.444367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595478,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":144,"num_read_kb":126,"num_write":286,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377004,"kb_used_data":11172,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564820,"statfs":{"total":1151940427776,"available":1151554375680,"internally_reserved":0,"allocated":11440128,"data_stored":7084258,"data_compressed":108786,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5110,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.288398"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169834+0000","last_change":"2024-03-18T15:57:57.181734+0000","last_active":"2024-03-18T15:57:59.169834+0000","last_peered":"2024-03-18T15:57:59.169834+0000","last_clean":"2024-03-18T15:57:59.169834+0000","last_became_active":"2024-03-18T15:57:57.181321+0000","last_became_peered":"2024-03-18T15:57:57.181321+0000","last_unstale":"2024-03-18T15:57:59.169834+0000","last_undegraded":"2024-03-18T15:57:59.169834+0000","last_fullsized":"2024-03-18T15:57:59.169834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:14.921877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170588+0000","last_change":"2024-03-18T15:57:57.175040+0000","last_active":"2024-03-18T15:57:59.170588+0000","last_peered":"2024-03-18T15:57:59.170588+0000","last_clean":"2024-03-18T15:57:59.170588+0000","last_became_active":"2024-03-18T15:57:57.174721+0000","last_became_peered":"2024-03-18T15:57:57.174721+0000","last_unstale":"2024-03-18T15:57:59.170588+0000","last_undegraded":"2024-03-18T15:57:59.170588+0000","last_fullsized":"2024-03-18T15:57:59.170588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:30.844693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170468+0000","last_change":"2024-03-18T15:57:57.183409+0000","last_active":"2024-03-18T15:57:59.170468+0000","last_peered":"2024-03-18T15:57:59.170468+0000","last_clean":"2024-03-18T15:57:59.170468+0000","last_became_active":"2024-03-18T15:57:57.183089+0000","last_became_peered":"2024-03-18T15:57:57.183089+0000","last_unstale":"2024-03-18T15:57:59.170468+0000","last_undegraded":"2024-03-18T15:57:59.170468+0000","last_fullsized":"2024-03-18T15:57:59.170468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:25.702402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171710+0000","last_change":"2024-03-18T15:57:57.183104+0000","last_active":"2024-03-18T15:57:59.171710+0000","last_peered":"2024-03-18T15:57:59.171710+0000","last_clean":"2024-03-18T15:57:59.171710+0000","last_became_active":"2024-03-18T15:57:57.182750+0000","last_became_peered":"2024-03-18T15:57:57.182750+0000","last_unstale":"2024-03-18T15:57:59.171710+0000","last_undegraded":"2024-03-18T15:57:59.171710+0000","last_fullsized":"2024-03-18T15:57:59.171710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:25.369309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167428+0000","last_change":"2024-03-18T15:57:57.195272+0000","last_active":"2024-03-18T15:57:59.167428+0000","last_peered":"2024-03-18T15:57:59.167428+0000","last_clean":"2024-03-18T15:57:59.167428+0000","last_became_active":"2024-03-18T15:57:57.192605+0000","last_became_peered":"2024-03-18T15:57:57.192605+0000","last_unstale":"2024-03-18T15:57:59.167428+0000","last_undegraded":"2024-03-18T15:57:59.167428+0000","last_fullsized":"2024-03-18T15:57:59.167428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:05.719973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170423+0000","last_change":"2024-03-18T15:57:57.177918+0000","last_active":"2024-03-18T15:57:59.170423+0000","last_peered":"2024-03-18T15:57:59.170423+0000","last_clean":"2024-03-18T15:57:59.170423+0000","last_became_active":"2024-03-18T15:57:57.177551+0000","last_became_peered":"2024-03-18T15:57:57.177551+0000","last_unstale":"2024-03-18T15:57:59.170423+0000","last_undegraded":"2024-03-18T15:57:59.170423+0000","last_fullsized":"2024-03-18T15:57:59.170423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:15.033755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170101+0000","last_change":"2024-03-18T15:57:57.179306+0000","last_active":"2024-03-18T15:57:59.170101+0000","last_peered":"2024-03-18T15:57:59.170101+0000","last_clean":"2024-03-18T15:57:59.170101+0000","last_became_active":"2024-03-18T15:57:57.178955+0000","last_became_peered":"2024-03-18T15:57:57.178955+0000","last_unstale":"2024-03-18T15:57:59.170101+0000","last_undegraded":"2024-03-18T15:57:59.170101+0000","last_fullsized":"2024-03-18T15:57:59.170101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:47.343910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.172976+0000","last_change":"2024-03-18T15:57:57.182548+0000","last_active":"2024-03-18T15:57:58.172976+0000","last_peered":"2024-03-18T15:57:58.172976+0000","last_clean":"2024-03-18T15:57:58.172976+0000","last_became_active":"2024-03-18T15:57:57.180297+0000","last_became_peered":"2024-03-18T15:57:57.180297+0000","last_unstale":"2024-03-18T15:57:58.172976+0000","last_undegraded":"2024-03-18T15:57:58.172976+0000","last_fullsized":"2024-03-18T15:57:58.172976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:47.392449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174048+0000","last_change":"2024-03-18T15:57:57.190203+0000","last_active":"2024-03-18T15:57:58.174048+0000","last_peered":"2024-03-18T15:57:58.174048+0000","last_clean":"2024-03-18T15:57:58.174048+0000","last_became_active":"2024-03-18T15:57:57.189850+0000","last_became_peered":"2024-03-18T15:57:57.189850+0000","last_unstale":"2024-03-18T15:57:58.174048+0000","last_undegraded":"2024-03-18T15:57:58.174048+0000","last_fullsized":"2024-03-18T15:57:58.174048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:46.274479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171061+0000","last_change":"2024-03-18T15:57:57.171053+0000","last_active":"2024-03-18T15:57:59.171061+0000","last_peered":"2024-03-18T15:57:59.171061+0000","last_clean":"2024-03-18T15:57:59.171061+0000","last_became_active":"2024-03-18T15:57:57.170727+0000","last_became_peered":"2024-03-18T15:57:57.170727+0000","last_unstale":"2024-03-18T15:57:59.171061+0000","last_undegraded":"2024-03-18T15:57:59.171061+0000","last_fullsized":"2024-03-18T15:57:59.171061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170789+0000","last_change":"2024-03-18T15:57:57.184219+0000","last_active":"2024-03-18T15:57:59.170789+0000","last_peered":"2024-03-18T15:57:59.170789+0000","last_clean":"2024-03-18T15:57:59.170789+0000","last_became_active":"2024-03-18T15:57:57.183575+0000","last_became_peered":"2024-03-18T15:57:57.183575+0000","last_unstale":"2024-03-18T15:57:59.170789+0000","last_undegraded":"2024-03-18T15:57:59.170789+0000","last_fullsized":"2024-03-18T15:57:59.170789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:59.171634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171637+0000","last_change":"2024-03-18T15:57:57.185462+0000","last_active":"2024-03-18T15:57:59.171637+0000","last_peered":"2024-03-18T15:57:59.171637+0000","last_clean":"2024-03-18T15:57:59.171637+0000","last_became_active":"2024-03-18T15:57:57.185136+0000","last_became_peered":"2024-03-18T15:57:57.185136+0000","last_unstale":"2024-03-18T15:57:59.171637+0000","last_undegraded":"2024-03-18T15:57:59.171637+0000","last_fullsized":"2024-03-18T15:57:59.171637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:36.579297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169787+0000","last_change":"2024-03-18T15:57:57.176638+0000","last_active":"2024-03-18T15:57:59.169787+0000","last_peered":"2024-03-18T15:57:59.169787+0000","last_clean":"2024-03-18T15:57:59.169787+0000","last_became_active":"2024-03-18T15:57:57.176327+0000","last_became_peered":"2024-03-18T15:57:57.176327+0000","last_unstale":"2024-03-18T15:57:59.169787+0000","last_undegraded":"2024-03-18T15:57:59.169787+0000","last_fullsized":"2024-03-18T15:57:59.169787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:18.115306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170129+0000","last_change":"2024-03-18T15:57:57.172258+0000","last_active":"2024-03-18T15:57:59.170129+0000","last_peered":"2024-03-18T15:57:59.170129+0000","last_clean":"2024-03-18T15:57:59.170129+0000","last_became_active":"2024-03-18T15:57:57.171916+0000","last_became_peered":"2024-03-18T15:57:57.171916+0000","last_unstale":"2024-03-18T15:57:59.170129+0000","last_undegraded":"2024-03-18T15:57:59.170129+0000","last_fullsized":"2024-03-18T15:57:59.170129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:24.467000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169788+0000","last_change":"2024-03-18T15:57:57.173523+0000","last_active":"2024-03-18T15:57:59.169788+0000","last_peered":"2024-03-18T15:57:59.169788+0000","last_clean":"2024-03-18T15:57:59.169788+0000","last_became_active":"2024-03-18T15:57:57.173203+0000","last_became_peered":"2024-03-18T15:57:57.173203+0000","last_unstale":"2024-03-18T15:57:59.169788+0000","last_undegraded":"2024-03-18T15:57:59.169788+0000","last_fullsized":"2024-03-18T15:57:59.169788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:31.383153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173614+0000","last_change":"2024-03-18T15:57:57.185214+0000","last_active":"2024-03-18T15:57:58.173614+0000","last_peered":"2024-03-18T15:57:58.173614+0000","last_clean":"2024-03-18T15:57:58.173614+0000","last_became_active":"2024-03-18T15:57:57.183794+0000","last_became_peered":"2024-03-18T15:57:57.183794+0000","last_unstale":"2024-03-18T15:57:58.173614+0000","last_undegraded":"2024-03-18T15:57:58.173614+0000","last_fullsized":"2024-03-18T15:57:58.173614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:45.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169816+0000","last_change":"2024-03-18T15:57:57.183349+0000","last_active":"2024-03-18T15:57:59.169816+0000","last_peered":"2024-03-18T15:57:59.169816+0000","last_clean":"2024-03-18T15:57:59.169816+0000","last_became_active":"2024-03-18T15:57:57.182127+0000","last_became_peered":"2024-03-18T15:57:57.182127+0000","last_unstale":"2024-03-18T15:57:59.169816+0000","last_undegraded":"2024-03-18T15:57:59.169816+0000","last_fullsized":"2024-03-18T15:57:59.169816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:17.229945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594614+0000","last_change":"2024-03-18T15:57:57.182112+0000","last_active":"2024-03-18T15:57:58.594614+0000","last_peered":"2024-03-18T15:57:58.594614+0000","last_clean":"2024-03-18T15:57:58.594614+0000","last_became_active":"2024-03-18T15:57:57.181695+0000","last_became_peered":"2024-03-18T15:57:57.181695+0000","last_unstale":"2024-03-18T15:57:58.594614+0000","last_undegraded":"2024-03-18T15:57:58.594614+0000","last_fullsized":"2024-03-18T15:57:58.594614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:20.402597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350248+0000","last_change":"2024-03-18T15:57:57.182187+0000","last_active":"2024-03-18T15:57:58.350248+0000","last_peered":"2024-03-18T15:57:58.350248+0000","last_clean":"2024-03-18T15:57:58.350248+0000","last_became_active":"2024-03-18T15:57:57.181837+0000","last_became_peered":"2024-03-18T15:57:57.181837+0000","last_unstale":"2024-03-18T15:57:58.350248+0000","last_undegraded":"2024-03-18T15:57:58.350248+0000","last_fullsized":"2024-03-18T15:57:58.350248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:37:32.225077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594440+0000","last_change":"2024-03-18T15:57:56.171691+0000","last_active":"2024-03-18T15:57:58.594440+0000","last_peered":"2024-03-18T15:57:58.594440+0000","last_clean":"2024-03-18T15:57:58.594440+0000","last_became_active":"2024-03-18T15:57:56.171241+0000","last_became_peered":"2024-03-18T15:57:56.171241+0000","last_unstale":"2024-03-18T15:57:58.594440+0000","last_undegraded":"2024-03-18T15:57:58.594440+0000","last_fullsized":"2024-03-18T15:57:58.594440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:50.411828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173550+0000","last_change":"2024-03-18T15:57:57.191757+0000","last_active":"2024-03-18T15:57:58.173550+0000","last_peered":"2024-03-18T15:57:58.173550+0000","last_clean":"2024-03-18T15:57:58.173550+0000","last_became_active":"2024-03-18T15:57:57.191474+0000","last_became_peered":"2024-03-18T15:57:57.191474+0000","last_unstale":"2024-03-18T15:57:58.173550+0000","last_undegraded":"2024-03-18T15:57:58.173550+0000","last_fullsized":"2024-03-18T15:57:58.173550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:18.430632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173709+0000","last_change":"2024-03-18T15:57:56.176784+0000","last_active":"2024-03-18T15:57:58.173709+0000","last_peered":"2024-03-18T15:57:58.173709+0000","last_clean":"2024-03-18T15:57:58.173709+0000","last_became_active":"2024-03-18T15:57:56.176468+0000","last_became_peered":"2024-03-18T15:57:56.176468+0000","last_unstale":"2024-03-18T15:57:58.173709+0000","last_undegraded":"2024-03-18T15:57:58.173709+0000","last_fullsized":"2024-03-18T15:57:58.173709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:50.941724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170451+0000","last_change":"2024-03-18T15:57:57.183492+0000","last_active":"2024-03-18T15:57:59.170451+0000","last_peered":"2024-03-18T15:57:59.170451+0000","last_clean":"2024-03-18T15:57:59.170451+0000","last_became_active":"2024-03-18T15:57:57.183156+0000","last_became_peered":"2024-03-18T15:57:57.183156+0000","last_unstale":"2024-03-18T15:57:59.170451+0000","last_undegraded":"2024-03-18T15:57:59.170451+0000","last_fullsized":"2024-03-18T15:57:59.170451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:41.441788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167791+0000","last_change":"2024-03-18T15:57:56.176614+0000","last_active":"2024-03-18T15:57:59.167791+0000","last_peered":"2024-03-18T15:57:59.167791+0000","last_clean":"2024-03-18T15:57:59.167791+0000","last_became_active":"2024-03-18T15:57:56.176245+0000","last_became_peered":"2024-03-18T15:57:56.176245+0000","last_unstale":"2024-03-18T15:57:59.167791+0000","last_undegraded":"2024-03-18T15:57:59.167791+0000","last_fullsized":"2024-03-18T15:57:59.167791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:11.647223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173783+0000","last_change":"2024-03-18T15:57:57.180588+0000","last_active":"2024-03-18T15:57:58.173783+0000","last_peered":"2024-03-18T15:57:58.173783+0000","last_clean":"2024-03-18T15:57:58.173783+0000","last_became_active":"2024-03-18T15:57:57.180245+0000","last_became_peered":"2024-03-18T15:57:57.180245+0000","last_unstale":"2024-03-18T15:57:58.173783+0000","last_undegraded":"2024-03-18T15:57:58.173783+0000","last_fullsized":"2024-03-18T15:57:58.173783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:38.095406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167593+0000","last_change":"2024-03-18T15:57:56.179815+0000","last_active":"2024-03-18T15:57:59.167593+0000","last_peered":"2024-03-18T15:57:59.167593+0000","last_clean":"2024-03-18T15:57:59.167593+0000","last_became_active":"2024-03-18T15:57:56.179495+0000","last_became_peered":"2024-03-18T15:57:56.179495+0000","last_unstale":"2024-03-18T15:57:59.167593+0000","last_undegraded":"2024-03-18T15:57:59.167593+0000","last_fullsized":"2024-03-18T15:57:59.167593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:17.876766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174164+0000","last_change":"2024-03-18T15:57:57.180021+0000","last_active":"2024-03-18T15:57:58.174164+0000","last_peered":"2024-03-18T15:57:58.174164+0000","last_clean":"2024-03-18T15:57:58.174164+0000","last_became_active":"2024-03-18T15:57:57.179642+0000","last_became_peered":"2024-03-18T15:57:57.179642+0000","last_unstale":"2024-03-18T15:57:58.174164+0000","last_undegraded":"2024-03-18T15:57:58.174164+0000","last_fullsized":"2024-03-18T15:57:58.174164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:36.573584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:01.202331+0000","last_change":"2024-03-18T15:57:56.178907+0000","last_active":"2024-03-18T15:58:01.202331+0000","last_peered":"2024-03-18T15:58:01.202331+0000","last_clean":"2024-03-18T15:58:01.202331+0000","last_became_active":"2024-03-18T15:57:56.178420+0000","last_became_peered":"2024-03-18T15:57:56.178420+0000","last_unstale":"2024-03-18T15:58:01.202331+0000","last_undegraded":"2024-03-18T15:58:01.202331+0000","last_fullsized":"2024-03-18T15:58:01.202331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:24:45.513178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167581+0000","last_change":"2024-03-18T15:57:57.187726+0000","last_active":"2024-03-18T15:57:59.167581+0000","last_peered":"2024-03-18T15:57:59.167581+0000","last_clean":"2024-03-18T15:57:59.167581+0000","last_became_active":"2024-03-18T15:57:57.187383+0000","last_became_peered":"2024-03-18T15:57:57.187383+0000","last_unstale":"2024-03-18T15:57:59.167581+0000","last_undegraded":"2024-03-18T15:57:59.167581+0000","last_fullsized":"2024-03-18T15:57:59.167581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:23.134839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171337+0000","last_change":"2024-03-18T15:57:56.175120+0000","last_active":"2024-03-18T15:57:59.171337+0000","last_peered":"2024-03-18T15:57:59.171337+0000","last_clean":"2024-03-18T15:57:59.171337+0000","last_became_active":"2024-03-18T15:57:56.174554+0000","last_became_peered":"2024-03-18T15:57:56.174554+0000","last_unstale":"2024-03-18T15:57:59.171337+0000","last_undegraded":"2024-03-18T15:57:59.171337+0000","last_fullsized":"2024-03-18T15:57:59.171337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:16.894313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174247+0000","last_change":"2024-03-18T15:57:57.192612+0000","last_active":"2024-03-18T15:57:58.174247+0000","last_peered":"2024-03-18T15:57:58.174247+0000","last_clean":"2024-03-18T15:57:58.174247+0000","last_became_active":"2024-03-18T15:57:57.192284+0000","last_became_peered":"2024-03-18T15:57:57.192284+0000","last_unstale":"2024-03-18T15:57:58.174247+0000","last_undegraded":"2024-03-18T15:57:58.174247+0000","last_fullsized":"2024-03-18T15:57:58.174247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:53.597362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198580+0000","last_change":"2024-03-18T15:57:56.165164+0000","last_active":"2024-03-18T15:58:00.198580+0000","last_peered":"2024-03-18T15:58:00.198580+0000","last_clean":"2024-03-18T15:58:00.198580+0000","last_became_active":"2024-03-18T15:57:56.164844+0000","last_became_peered":"2024-03-18T15:57:56.164844+0000","last_unstale":"2024-03-18T15:58:00.198580+0000","last_undegraded":"2024-03-18T15:58:00.198580+0000","last_fullsized":"2024-03-18T15:58:00.198580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:36.754823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173060+0000","last_change":"2024-03-18T15:57:57.180176+0000","last_active":"2024-03-18T15:57:58.173060+0000","last_peered":"2024-03-18T15:57:58.173060+0000","last_clean":"2024-03-18T15:57:58.173060+0000","last_became_active":"2024-03-18T15:57:57.179832+0000","last_became_peered":"2024-03-18T15:57:57.179832+0000","last_unstale":"2024-03-18T15:57:58.173060+0000","last_undegraded":"2024-03-18T15:57:58.173060+0000","last_fullsized":"2024-03-18T15:57:58.173060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:32.249126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173739+0000","last_change":"2024-03-18T15:57:56.175404+0000","last_active":"2024-03-18T15:57:58.173739+0000","last_peered":"2024-03-18T15:57:58.173739+0000","last_clean":"2024-03-18T15:57:58.173739+0000","last_became_active":"2024-03-18T15:57:56.174877+0000","last_became_peered":"2024-03-18T15:57:56.174877+0000","last_unstale":"2024-03-18T15:57:58.173739+0000","last_undegraded":"2024-03-18T15:57:58.173739+0000","last_fullsized":"2024-03-18T15:57:58.173739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:39.536240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594506+0000","last_change":"2024-03-18T15:57:57.178213+0000","last_active":"2024-03-18T15:57:58.594506+0000","last_peered":"2024-03-18T15:57:58.594506+0000","last_clean":"2024-03-18T15:57:58.594506+0000","last_became_active":"2024-03-18T15:57:57.177863+0000","last_became_peered":"2024-03-18T15:57:57.177863+0000","last_unstale":"2024-03-18T15:57:58.594506+0000","last_undegraded":"2024-03-18T15:57:58.594506+0000","last_fullsized":"2024-03-18T15:57:58.594506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:42.988890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190571+0000","last_change":"2024-03-18T15:57:56.172506+0000","last_active":"2024-03-18T15:57:59.190571+0000","last_peered":"2024-03-18T15:57:59.190571+0000","last_clean":"2024-03-18T15:57:59.190571+0000","last_became_active":"2024-03-18T15:57:56.172153+0000","last_became_peered":"2024-03-18T15:57:56.172153+0000","last_unstale":"2024-03-18T15:57:59.190571+0000","last_undegraded":"2024-03-18T15:57:59.190571+0000","last_fullsized":"2024-03-18T15:57:59.190571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:30.563167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173965+0000","last_change":"2024-03-18T15:57:57.181506+0000","last_active":"2024-03-18T15:57:58.173965+0000","last_peered":"2024-03-18T15:57:58.173965+0000","last_clean":"2024-03-18T15:57:58.173965+0000","last_became_active":"2024-03-18T15:57:57.181139+0000","last_became_peered":"2024-03-18T15:57:57.181139+0000","last_unstale":"2024-03-18T15:57:58.173965+0000","last_undegraded":"2024-03-18T15:57:58.173965+0000","last_fullsized":"2024-03-18T15:57:58.173965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:40.239719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188980+0000","last_change":"2024-03-18T15:57:56.170895+0000","last_active":"2024-03-18T15:57:59.188980+0000","last_peered":"2024-03-18T15:57:59.188980+0000","last_clean":"2024-03-18T15:57:59.188980+0000","last_became_active":"2024-03-18T15:57:56.170394+0000","last_became_peered":"2024-03-18T15:57:56.170394+0000","last_unstale":"2024-03-18T15:57:59.188980+0000","last_undegraded":"2024-03-18T15:57:59.188980+0000","last_fullsized":"2024-03-18T15:57:59.188980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.915841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170192+0000","last_change":"2024-03-18T15:57:57.186766+0000","last_active":"2024-03-18T15:57:59.170192+0000","last_peered":"2024-03-18T15:57:59.170192+0000","last_clean":"2024-03-18T15:57:59.170192+0000","last_became_active":"2024-03-18T15:57:57.186453+0000","last_became_peered":"2024-03-18T15:57:57.186453+0000","last_unstale":"2024-03-18T15:57:59.170192+0000","last_undegraded":"2024-03-18T15:57:59.170192+0000","last_fullsized":"2024-03-18T15:57:59.170192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:19.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188521+0000","last_change":"2024-03-18T15:57:56.174536+0000","last_active":"2024-03-18T15:57:59.188521+0000","last_peered":"2024-03-18T15:57:59.188521+0000","last_clean":"2024-03-18T15:57:59.188521+0000","last_became_active":"2024-03-18T15:57:56.174216+0000","last_became_peered":"2024-03-18T15:57:56.174216+0000","last_unstale":"2024-03-18T15:57:59.188521+0000","last_undegraded":"2024-03-18T15:57:59.188521+0000","last_fullsized":"2024-03-18T15:57:59.188521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:50.324174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173861+0000","last_change":"2024-03-18T15:57:57.178511+0000","last_active":"2024-03-18T15:57:58.173861+0000","last_peered":"2024-03-18T15:57:58.173861+0000","last_clean":"2024-03-18T15:57:58.173861+0000","last_became_active":"2024-03-18T15:57:57.178117+0000","last_became_peered":"2024-03-18T15:57:57.178117+0000","last_unstale":"2024-03-18T15:57:58.173861+0000","last_undegraded":"2024-03-18T15:57:58.173861+0000","last_fullsized":"2024-03-18T15:57:58.173861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:52.891086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.189683+0000","last_change":"2024-03-18T15:57:56.173372+0000","last_active":"2024-03-18T15:57:59.189683+0000","last_peered":"2024-03-18T15:57:59.189683+0000","last_clean":"2024-03-18T15:57:59.189683+0000","last_became_active":"2024-03-18T15:57:56.173049+0000","last_became_peered":"2024-03-18T15:57:56.173049+0000","last_unstale":"2024-03-18T15:57:59.189683+0000","last_undegraded":"2024-03-18T15:57:59.189683+0000","last_fullsized":"2024-03-18T15:57:59.189683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:01.655747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167646+0000","last_change":"2024-03-18T15:57:57.183725+0000","last_active":"2024-03-18T15:57:59.167646+0000","last_peered":"2024-03-18T15:57:59.167646+0000","last_clean":"2024-03-18T15:57:59.167646+0000","last_became_active":"2024-03-18T15:57:57.183391+0000","last_became_peered":"2024-03-18T15:57:57.183391+0000","last_unstale":"2024-03-18T15:57:59.167646+0000","last_undegraded":"2024-03-18T15:57:59.167646+0000","last_fullsized":"2024-03-18T15:57:59.167646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:42.945519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198950+0000","last_change":"2024-03-18T15:57:56.179534+0000","last_active":"2024-03-18T15:58:00.198950+0000","last_peered":"2024-03-18T15:58:00.198950+0000","last_clean":"2024-03-18T15:58:00.198950+0000","last_became_active":"2024-03-18T15:57:56.179217+0000","last_became_peered":"2024-03-18T15:57:56.179217+0000","last_unstale":"2024-03-18T15:58:00.198950+0000","last_undegraded":"2024-03-18T15:58:00.198950+0000","last_fullsized":"2024-03-18T15:58:00.198950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:19.176423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169660+0000","last_change":"2024-03-18T15:57:57.188958+0000","last_active":"2024-03-18T15:57:59.169660+0000","last_peered":"2024-03-18T15:57:59.169660+0000","last_clean":"2024-03-18T15:57:59.169660+0000","last_became_active":"2024-03-18T15:57:57.188586+0000","last_became_peered":"2024-03-18T15:57:57.188586+0000","last_unstale":"2024-03-18T15:57:59.169660+0000","last_undegraded":"2024-03-18T15:57:59.169660+0000","last_fullsized":"2024-03-18T15:57:59.169660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:48.122539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188281+0000","last_change":"2024-03-18T15:57:56.180530+0000","last_active":"2024-03-18T15:57:59.188281+0000","last_peered":"2024-03-18T15:57:59.188281+0000","last_clean":"2024-03-18T15:57:59.188281+0000","last_became_active":"2024-03-18T15:57:56.180019+0000","last_became_peered":"2024-03-18T15:57:56.180019+0000","last_unstale":"2024-03-18T15:57:59.188281+0000","last_undegraded":"2024-03-18T15:57:59.188281+0000","last_fullsized":"2024-03-18T15:57:59.188281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:04.119730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171093+0000","last_change":"2024-03-18T15:57:57.175201+0000","last_active":"2024-03-18T15:57:59.171093+0000","last_peered":"2024-03-18T15:57:59.171093+0000","last_clean":"2024-03-18T15:57:59.171093+0000","last_became_active":"2024-03-18T15:57:57.174878+0000","last_became_peered":"2024-03-18T15:57:57.174878+0000","last_unstale":"2024-03-18T15:57:59.171093+0000","last_undegraded":"2024-03-18T15:57:59.171093+0000","last_fullsized":"2024-03-18T15:57:59.171093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:43.545594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198179+0000","last_change":"2024-03-18T15:57:56.172587+0000","last_active":"2024-03-18T15:58:00.198179+0000","last_peered":"2024-03-18T15:58:00.198179+0000","last_clean":"2024-03-18T15:58:00.198179+0000","last_became_active":"2024-03-18T15:57:56.172246+0000","last_became_peered":"2024-03-18T15:57:56.172246+0000","last_unstale":"2024-03-18T15:58:00.198179+0000","last_undegraded":"2024-03-18T15:58:00.198179+0000","last_fullsized":"2024-03-18T15:58:00.198179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.543059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169907+0000","last_change":"2024-03-18T15:57:57.181899+0000","last_active":"2024-03-18T15:57:59.169907+0000","last_peered":"2024-03-18T15:57:59.169907+0000","last_clean":"2024-03-18T15:57:59.169907+0000","last_became_active":"2024-03-18T15:57:57.181559+0000","last_became_peered":"2024-03-18T15:57:57.181559+0000","last_unstale":"2024-03-18T15:57:59.169907+0000","last_undegraded":"2024-03-18T15:57:59.169907+0000","last_fullsized":"2024-03-18T15:57:59.169907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:17.504153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190849+0000","last_change":"2024-03-18T15:57:56.179375+0000","last_active":"2024-03-18T15:57:59.190849+0000","last_peered":"2024-03-18T15:57:59.190849+0000","last_clean":"2024-03-18T15:57:59.190849+0000","last_became_active":"2024-03-18T15:57:56.178990+0000","last_became_peered":"2024-03-18T15:57:56.178990+0000","last_unstale":"2024-03-18T15:57:59.190849+0000","last_undegraded":"2024-03-18T15:57:59.190849+0000","last_fullsized":"2024-03-18T15:57:59.190849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:36.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173781+0000","last_change":"2024-03-18T15:57:57.188135+0000","last_active":"2024-03-18T15:57:58.173781+0000","last_peered":"2024-03-18T15:57:58.173781+0000","last_clean":"2024-03-18T15:57:58.173781+0000","last_became_active":"2024-03-18T15:57:57.187811+0000","last_became_peered":"2024-03-18T15:57:57.187811+0000","last_unstale":"2024-03-18T15:57:58.173781+0000","last_undegraded":"2024-03-18T15:57:58.173781+0000","last_fullsized":"2024-03-18T15:57:58.173781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:11.235420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167711+0000","last_change":"2024-03-18T15:57:56.181774+0000","last_active":"2024-03-18T15:57:59.167711+0000","last_peered":"2024-03-18T15:57:59.167711+0000","last_clean":"2024-03-18T15:57:59.167711+0000","last_became_active":"2024-03-18T15:57:56.181360+0000","last_became_peered":"2024-03-18T15:57:56.181360+0000","last_unstale":"2024-03-18T15:57:59.167711+0000","last_undegraded":"2024-03-18T15:57:59.167711+0000","last_fullsized":"2024-03-18T15:57:59.167711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:08.367151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169680+0000","last_change":"2024-03-18T15:57:57.188960+0000","last_active":"2024-03-18T15:57:59.169680+0000","last_peered":"2024-03-18T15:57:59.169680+0000","last_clean":"2024-03-18T15:57:59.169680+0000","last_became_active":"2024-03-18T15:57:57.187528+0000","last_became_peered":"2024-03-18T15:57:57.187528+0000","last_unstale":"2024-03-18T15:57:59.169680+0000","last_undegraded":"2024-03-18T15:57:59.169680+0000","last_fullsized":"2024-03-18T15:57:59.169680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:29.259977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594450+0000","last_change":"2024-03-18T15:57:56.174055+0000","last_active":"2024-03-18T15:57:58.594450+0000","last_peered":"2024-03-18T15:57:58.594450+0000","last_clean":"2024-03-18T15:57:58.594450+0000","last_became_active":"2024-03-18T15:57:56.173736+0000","last_became_peered":"2024-03-18T15:57:56.173736+0000","last_unstale":"2024-03-18T15:57:58.594450+0000","last_undegraded":"2024-03-18T15:57:58.594450+0000","last_fullsized":"2024-03-18T15:57:58.594450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:00.319049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170834+0000","last_change":"2024-03-18T15:57:57.189238+0000","last_active":"2024-03-18T15:57:59.170834+0000","last_peered":"2024-03-18T15:57:59.170834+0000","last_clean":"2024-03-18T15:57:59.170834+0000","last_became_active":"2024-03-18T15:57:57.188900+0000","last_became_peered":"2024-03-18T15:57:57.188900+0000","last_unstale":"2024-03-18T15:57:59.170834+0000","last_undegraded":"2024-03-18T15:57:59.170834+0000","last_fullsized":"2024-03-18T15:57:59.170834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:16.810976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170265+0000","last_change":"2024-03-18T15:57:56.178765+0000","last_active":"2024-03-18T15:57:59.170265+0000","last_peered":"2024-03-18T15:57:59.170265+0000","last_clean":"2024-03-18T15:57:59.170265+0000","last_became_active":"2024-03-18T15:57:56.178399+0000","last_became_peered":"2024-03-18T15:57:56.178399+0000","last_unstale":"2024-03-18T15:57:59.170265+0000","last_undegraded":"2024-03-18T15:57:59.170265+0000","last_fullsized":"2024-03-18T15:57:59.170265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:46:07.844453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171300+0000","last_change":"2024-03-18T15:57:57.173968+0000","last_active":"2024-03-18T15:57:59.171300+0000","last_peered":"2024-03-18T15:57:59.171300+0000","last_clean":"2024-03-18T15:57:59.171300+0000","last_became_active":"2024-03-18T15:57:57.173605+0000","last_became_peered":"2024-03-18T15:57:57.173605+0000","last_unstale":"2024-03-18T15:57:59.171300+0000","last_undegraded":"2024-03-18T15:57:59.171300+0000","last_fullsized":"2024-03-18T15:57:59.171300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:34.676393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174385+0000","last_change":"2024-03-18T15:57:57.186561+0000","last_active":"2024-03-18T15:57:58.174385+0000","last_peered":"2024-03-18T15:57:58.174385+0000","last_clean":"2024-03-18T15:57:58.174385+0000","last_became_active":"2024-03-18T15:57:57.186260+0000","last_became_peered":"2024-03-18T15:57:57.186260+0000","last_unstale":"2024-03-18T15:57:58.174385+0000","last_undegraded":"2024-03-18T15:57:58.174385+0000","last_fullsized":"2024-03-18T15:57:58.174385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:54:06.646480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.200785+0000","last_change":"2024-03-18T15:57:56.164175+0000","last_active":"2024-03-18T15:58:00.200785+0000","last_peered":"2024-03-18T15:58:00.200785+0000","last_clean":"2024-03-18T15:58:00.200785+0000","last_became_active":"2024-03-18T15:57:56.163724+0000","last_became_peered":"2024-03-18T15:57:56.163724+0000","last_unstale":"2024-03-18T15:58:00.200785+0000","last_undegraded":"2024-03-18T15:58:00.200785+0000","last_fullsized":"2024-03-18T15:58:00.200785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:48:53.596211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173257+0000","last_change":"2024-03-18T15:57:57.191507+0000","last_active":"2024-03-18T15:57:58.173257+0000","last_peered":"2024-03-18T15:57:58.173257+0000","last_clean":"2024-03-18T15:57:58.173257+0000","last_became_active":"2024-03-18T15:57:57.191182+0000","last_became_peered":"2024-03-18T15:57:57.191182+0000","last_unstale":"2024-03-18T15:57:58.173257+0000","last_undegraded":"2024-03-18T15:57:58.173257+0000","last_fullsized":"2024-03-18T15:57:58.173257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:39.878675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.188475+0000","last_change":"2024-03-18T15:57:56.169490+0000","last_active":"2024-03-18T15:57:59.188475+0000","last_peered":"2024-03-18T15:57:59.188475+0000","last_clean":"2024-03-18T15:57:59.188475+0000","last_became_active":"2024-03-18T15:57:56.169107+0000","last_became_peered":"2024-03-18T15:57:56.169107+0000","last_unstale":"2024-03-18T15:57:59.188475+0000","last_undegraded":"2024-03-18T15:57:59.188475+0000","last_fullsized":"2024-03-18T15:57:59.188475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:59.722909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169906+0000","last_change":"2024-03-18T15:56:43.270716+0000","last_active":"2024-03-18T15:57:59.169906+0000","last_peered":"2024-03-18T15:57:59.169906+0000","last_clean":"2024-03-18T15:57:59.169906+0000","last_became_active":"2024-03-18T15:56:43.270316+0000","last_became_peered":"2024-03-18T15:56:43.270316+0000","last_unstale":"2024-03-18T15:57:59.169906+0000","last_undegraded":"2024-03-18T15:57:59.169906+0000","last_fullsized":"2024-03-18T15:57:59.169906+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_clean_scrub_stamp":"2024-03-18T15:53:17.916048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:00.779948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169444+0000","last_change":"2024-03-18T15:57:57.172265+0000","last_active":"2024-03-18T15:57:59.169444+0000","last_peered":"2024-03-18T15:57:59.169444+0000","last_clean":"2024-03-18T15:57:59.169444+0000","last_became_active":"2024-03-18T15:57:57.171962+0000","last_became_peered":"2024-03-18T15:57:57.171962+0000","last_unstale":"2024-03-18T15:57:59.169444+0000","last_undegraded":"2024-03-18T15:57:59.169444+0000","last_fullsized":"2024-03-18T15:57:59.169444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:41:44.402012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.198203+0000","last_change":"2024-03-18T15:57:56.176528+0000","last_active":"2024-03-18T15:58:00.198203+0000","last_peered":"2024-03-18T15:58:00.198203+0000","last_clean":"2024-03-18T15:58:00.198203+0000","last_became_active":"2024-03-18T15:57:56.176137+0000","last_became_peered":"2024-03-18T15:57:56.176137+0000","last_unstale":"2024-03-18T15:58:00.198203+0000","last_undegraded":"2024-03-18T15:58:00.198203+0000","last_fullsized":"2024-03-18T15:58:00.198203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:19:14.299323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190946+0000","last_change":"2024-03-18T15:57:56.178710+0000","last_active":"2024-03-18T15:57:59.190946+0000","last_peered":"2024-03-18T15:57:59.190946+0000","last_clean":"2024-03-18T15:57:59.190946+0000","last_became_active":"2024-03-18T15:57:56.178388+0000","last_became_peered":"2024-03-18T15:57:56.178388+0000","last_unstale":"2024-03-18T15:57:59.190946+0000","last_undegraded":"2024-03-18T15:57:59.190946+0000","last_fullsized":"2024-03-18T15:57:59.190946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:23.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170074+0000","last_change":"2024-03-18T15:57:57.168951+0000","last_active":"2024-03-18T15:57:59.170074+0000","last_peered":"2024-03-18T15:57:59.170074+0000","last_clean":"2024-03-18T15:57:59.170074+0000","last_became_active":"2024-03-18T15:57:57.168567+0000","last_became_peered":"2024-03-18T15:57:57.168567+0000","last_unstale":"2024-03-18T15:57:59.170074+0000","last_undegraded":"2024-03-18T15:57:59.170074+0000","last_fullsized":"2024-03-18T15:57:59.170074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:41.171254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.594622+0000","last_change":"2024-03-18T15:57:56.171665+0000","last_active":"2024-03-18T15:57:58.594622+0000","last_peered":"2024-03-18T15:57:58.594622+0000","last_clean":"2024-03-18T15:57:58.594622+0000","last_became_active":"2024-03-18T15:57:56.171282+0000","last_became_peered":"2024-03-18T15:57:56.171282+0000","last_unstale":"2024-03-18T15:57:58.594622+0000","last_undegraded":"2024-03-18T15:57:58.594622+0000","last_fullsized":"2024-03-18T15:57:58.594622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:03:27.523561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173508+0000","last_change":"2024-03-18T15:57:57.185647+0000","last_active":"2024-03-18T15:57:58.173508+0000","last_peered":"2024-03-18T15:57:58.173508+0000","last_clean":"2024-03-18T15:57:58.173508+0000","last_became_active":"2024-03-18T15:57:57.185352+0000","last_became_peered":"2024-03-18T15:57:57.185352+0000","last_unstale":"2024-03-18T15:57:58.173508+0000","last_undegraded":"2024-03-18T15:57:58.173508+0000","last_fullsized":"2024-03-18T15:57:58.173508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:38.247776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.186067+0000","last_change":"2024-03-18T15:57:56.170878+0000","last_active":"2024-03-18T15:57:59.186067+0000","last_peered":"2024-03-18T15:57:59.186067+0000","last_clean":"2024-03-18T15:57:59.186067+0000","last_became_active":"2024-03-18T15:57:56.170558+0000","last_became_peered":"2024-03-18T15:57:56.170558+0000","last_unstale":"2024-03-18T15:57:59.186067+0000","last_undegraded":"2024-03-18T15:57:59.186067+0000","last_fullsized":"2024-03-18T15:57:59.186067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:26:39.963384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170622+0000","last_change":"2024-03-18T15:57:57.181012+0000","last_active":"2024-03-18T15:57:59.170622+0000","last_peered":"2024-03-18T15:57:59.170622+0000","last_clean":"2024-03-18T15:57:59.170622+0000","last_became_active":"2024-03-18T15:57:57.180613+0000","last_became_peered":"2024-03-18T15:57:57.180613+0000","last_unstale":"2024-03-18T15:57:59.170622+0000","last_undegraded":"2024-03-18T15:57:59.170622+0000","last_fullsized":"2024-03-18T15:57:59.170622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:22.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190523+0000","last_change":"2024-03-18T15:57:56.171929+0000","last_active":"2024-03-18T15:57:59.190523+0000","last_peered":"2024-03-18T15:57:59.190523+0000","last_clean":"2024-03-18T15:57:59.190523+0000","last_became_active":"2024-03-18T15:57:56.171570+0000","last_became_peered":"2024-03-18T15:57:56.171570+0000","last_unstale":"2024-03-18T15:57:59.190523+0000","last_undegraded":"2024-03-18T15:57:59.190523+0000","last_fullsized":"2024-03-18T15:57:59.190523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:11.579956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.167389+0000","last_change":"2024-03-18T15:57:57.178118+0000","last_active":"2024-03-18T15:57:59.167389+0000","last_peered":"2024-03-18T15:57:59.167389+0000","last_clean":"2024-03-18T15:57:59.167389+0000","last_became_active":"2024-03-18T15:57:57.177764+0000","last_became_peered":"2024-03-18T15:57:57.177764+0000","last_unstale":"2024-03-18T15:57:59.167389+0000","last_undegraded":"2024-03-18T15:57:59.167389+0000","last_fullsized":"2024-03-18T15:57:59.167389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:07.519544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190597+0000","last_change":"2024-03-18T15:57:56.168485+0000","last_active":"2024-03-18T15:57:59.190597+0000","last_peered":"2024-03-18T15:57:59.190597+0000","last_clean":"2024-03-18T15:57:59.190597+0000","last_became_active":"2024-03-18T15:57:56.168124+0000","last_became_peered":"2024-03-18T15:57:56.168124+0000","last_unstale":"2024-03-18T15:57:59.190597+0000","last_undegraded":"2024-03-18T15:57:59.190597+0000","last_fullsized":"2024-03-18T15:57:59.190597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:09.465975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169540+0000","last_change":"2024-03-18T15:57:57.184781+0000","last_active":"2024-03-18T15:57:59.169540+0000","last_peered":"2024-03-18T15:57:59.169540+0000","last_clean":"2024-03-18T15:57:59.169540+0000","last_became_active":"2024-03-18T15:57:57.184431+0000","last_became_peered":"2024-03-18T15:57:57.184431+0000","last_unstale":"2024-03-18T15:57:59.169540+0000","last_undegraded":"2024-03-18T15:57:59.169540+0000","last_fullsized":"2024-03-18T15:57:59.169540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:32.945288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350312+0000","last_change":"2024-03-18T15:57:56.180295+0000","last_active":"2024-03-18T15:57:58.350312+0000","last_peered":"2024-03-18T15:57:58.350312+0000","last_clean":"2024-03-18T15:57:58.350312+0000","last_became_active":"2024-03-18T15:57:56.179956+0000","last_became_peered":"2024-03-18T15:57:56.179956+0000","last_unstale":"2024-03-18T15:57:58.350312+0000","last_undegraded":"2024-03-18T15:57:58.350312+0000","last_fullsized":"2024-03-18T15:57:58.350312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:49.664982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173501+0000","last_change":"2024-03-18T15:57:57.187421+0000","last_active":"2024-03-18T15:57:58.173501+0000","last_peered":"2024-03-18T15:57:58.173501+0000","last_clean":"2024-03-18T15:57:58.173501+0000","last_became_active":"2024-03-18T15:57:57.185978+0000","last_became_peered":"2024-03-18T15:57:57.185978+0000","last_unstale":"2024-03-18T15:57:58.173501+0000","last_undegraded":"2024-03-18T15:57:58.173501+0000","last_fullsized":"2024-03-18T15:57:58.173501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:07:59.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.190461+0000","last_change":"2024-03-18T15:57:56.178118+0000","last_active":"2024-03-18T15:57:59.190461+0000","last_peered":"2024-03-18T15:57:59.190461+0000","last_clean":"2024-03-18T15:57:59.190461+0000","last_became_active":"2024-03-18T15:57:56.177758+0000","last_became_peered":"2024-03-18T15:57:56.177758+0000","last_unstale":"2024-03-18T15:57:59.190461+0000","last_undegraded":"2024-03-18T15:57:59.190461+0000","last_fullsized":"2024-03-18T15:57:59.190461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:34.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170395+0000","last_change":"2024-03-18T15:57:57.182666+0000","last_active":"2024-03-18T15:57:59.170395+0000","last_peered":"2024-03-18T15:57:59.170395+0000","last_clean":"2024-03-18T15:57:59.170395+0000","last_became_active":"2024-03-18T15:57:57.182328+0000","last_became_peered":"2024-03-18T15:57:57.182328+0000","last_unstale":"2024-03-18T15:57:59.170395+0000","last_undegraded":"2024-03-18T15:57:59.170395+0000","last_fullsized":"2024-03-18T15:57:59.170395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:55.132441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:58:00.197362+0000","last_change":"2024-03-18T15:57:56.177101+0000","last_active":"2024-03-18T15:58:00.197362+0000","last_peered":"2024-03-18T15:58:00.197362+0000","last_clean":"2024-03-18T15:58:00.197362+0000","last_became_active":"2024-03-18T15:57:56.176734+0000","last_became_peered":"2024-03-18T15:57:56.176734+0000","last_unstale":"2024-03-18T15:58:00.197362+0000","last_undegraded":"2024-03-18T15:58:00.197362+0000","last_fullsized":"2024-03-18T15:58:00.197362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:55.406210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170968+0000","last_change":"2024-03-18T15:57:57.183892+0000","last_active":"2024-03-18T15:57:59.170968+0000","last_peered":"2024-03-18T15:57:59.170968+0000","last_clean":"2024-03-18T15:57:59.170968+0000","last_became_active":"2024-03-18T15:57:57.183121+0000","last_became_peered":"2024-03-18T15:57:57.183121+0000","last_unstale":"2024-03-18T15:57:59.170968+0000","last_undegraded":"2024-03-18T15:57:59.170968+0000","last_fullsized":"2024-03-18T15:57:59.170968+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:11:13.504169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.174105+0000","last_change":"2024-03-18T15:57:56.172479+0000","last_active":"2024-03-18T15:57:58.174105+0000","last_peered":"2024-03-18T15:57:58.174105+0000","last_clean":"2024-03-18T15:57:58.174105+0000","last_became_active":"2024-03-18T15:57:56.172120+0000","last_became_peered":"2024-03-18T15:57:56.172120+0000","last_unstale":"2024-03-18T15:57:58.174105+0000","last_undegraded":"2024-03-18T15:57:58.174105+0000","last_fullsized":"2024-03-18T15:57:58.174105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:15.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170482+0000","last_change":"2024-03-18T15:57:57.183674+0000","last_active":"2024-03-18T15:57:59.170482+0000","last_peered":"2024-03-18T15:57:59.170482+0000","last_clean":"2024-03-18T15:57:59.170482+0000","last_became_active":"2024-03-18T15:57:57.183270+0000","last_became_peered":"2024-03-18T15:57:57.183270+0000","last_unstale":"2024-03-18T15:57:59.170482+0000","last_undegraded":"2024-03-18T15:57:59.170482+0000","last_fullsized":"2024-03-18T15:57:59.170482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:43.258995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169735+0000","last_change":"2024-03-18T15:57:56.174497+0000","last_active":"2024-03-18T15:57:59.169735+0000","last_peered":"2024-03-18T15:57:59.169735+0000","last_clean":"2024-03-18T15:57:59.169735+0000","last_became_active":"2024-03-18T15:57:56.173862+0000","last_became_peered":"2024-03-18T15:57:56.173862+0000","last_unstale":"2024-03-18T15:57:59.169735+0000","last_undegraded":"2024-03-18T15:57:59.169735+0000","last_fullsized":"2024-03-18T15:57:59.169735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:00.362276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.171330+0000","last_change":"2024-03-18T15:57:57.179510+0000","last_active":"2024-03-18T15:57:59.171330+0000","last_peered":"2024-03-18T15:57:59.171330+0000","last_clean":"2024-03-18T15:57:59.171330+0000","last_became_active":"2024-03-18T15:57:57.179178+0000","last_became_peered":"2024-03-18T15:57:57.179178+0000","last_unstale":"2024-03-18T15:57:59.171330+0000","last_undegraded":"2024-03-18T15:57:59.171330+0000","last_fullsized":"2024-03-18T15:57:59.171330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:46.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169602+0000","last_change":"2024-03-18T15:57:57.186842+0000","last_active":"2024-03-18T15:57:59.169602+0000","last_peered":"2024-03-18T15:57:59.169602+0000","last_clean":"2024-03-18T15:57:59.169602+0000","last_became_active":"2024-03-18T15:57:57.186501+0000","last_became_peered":"2024-03-18T15:57:57.186501+0000","last_unstale":"2024-03-18T15:57:59.169602+0000","last_undegraded":"2024-03-18T15:57:59.169602+0000","last_fullsized":"2024-03-18T15:57:59.169602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:11.988494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350322+0000","last_change":"2024-03-18T15:57:57.177630+0000","last_active":"2024-03-18T15:57:58.350322+0000","last_peered":"2024-03-18T15:57:58.350322+0000","last_clean":"2024-03-18T15:57:58.350322+0000","last_became_active":"2024-03-18T15:57:57.177227+0000","last_became_peered":"2024-03-18T15:57:57.177227+0000","last_unstale":"2024-03-18T15:57:58.350322+0000","last_undegraded":"2024-03-18T15:57:58.350322+0000","last_fullsized":"2024-03-18T15:57:58.350322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:27.004129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350149+0000","last_change":"2024-03-18T15:57:57.189048+0000","last_active":"2024-03-18T15:57:58.350149+0000","last_peered":"2024-03-18T15:57:58.350149+0000","last_clean":"2024-03-18T15:57:58.350149+0000","last_became_active":"2024-03-18T15:57:57.188704+0000","last_became_peered":"2024-03-18T15:57:57.188704+0000","last_unstale":"2024-03-18T15:57:58.350149+0000","last_undegraded":"2024-03-18T15:57:58.350149+0000","last_fullsized":"2024-03-18T15:57:58.350149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:10:53.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.170546+0000","last_change":"2024-03-18T15:57:57.178611+0000","last_active":"2024-03-18T15:57:59.170546+0000","last_peered":"2024-03-18T15:57:59.170546+0000","last_clean":"2024-03-18T15:57:59.170546+0000","last_became_active":"2024-03-18T15:57:57.178305+0000","last_became_peered":"2024-03-18T15:57:57.178305+0000","last_unstale":"2024-03-18T15:57:59.170546+0000","last_undegraded":"2024-03-18T15:57:59.170546+0000","last_fullsized":"2024-03-18T15:57:59.170546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:51.925278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169899+0000","last_change":"2024-03-18T15:57:57.177928+0000","last_active":"2024-03-18T15:57:59.169899+0000","last_peered":"2024-03-18T15:57:59.169899+0000","last_clean":"2024-03-18T15:57:59.169899+0000","last_became_active":"2024-03-18T15:57:57.177364+0000","last_became_peered":"2024-03-18T15:57:57.177364+0000","last_unstale":"2024-03-18T15:57:59.169899+0000","last_undegraded":"2024-03-18T15:57:59.169899+0000","last_fullsized":"2024-03-18T15:57:59.169899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:59.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169705+0000","last_change":"2024-03-18T15:57:57.188080+0000","last_active":"2024-03-18T15:57:59.169705+0000","last_peered":"2024-03-18T15:57:59.169705+0000","last_clean":"2024-03-18T15:57:59.169705+0000","last_became_active":"2024-03-18T15:57:57.187767+0000","last_became_peered":"2024-03-18T15:57:57.187767+0000","last_unstale":"2024-03-18T15:57:59.169705+0000","last_undegraded":"2024-03-18T15:57:59.169705+0000","last_fullsized":"2024-03-18T15:57:59.169705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:26.290262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.350443+0000","last_change":"2024-03-18T15:57:57.178581+0000","last_active":"2024-03-18T15:57:58.350443+0000","last_peered":"2024-03-18T15:57:58.350443+0000","last_clean":"2024-03-18T15:57:58.350443+0000","last_became_active":"2024-03-18T15:57:57.178168+0000","last_became_peered":"2024-03-18T15:57:57.178168+0000","last_unstale":"2024-03-18T15:57:58.350443+0000","last_undegraded":"2024-03-18T15:57:58.350443+0000","last_fullsized":"2024-03-18T15:57:58.350443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:07:07.601959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173475+0000","last_change":"2024-03-18T15:57:57.188709+0000","last_active":"2024-03-18T15:57:58.173475+0000","last_peered":"2024-03-18T15:57:58.173475+0000","last_clean":"2024-03-18T15:57:58.173475+0000","last_became_active":"2024-03-18T15:57:57.187884+0000","last_became_peered":"2024-03-18T15:57:57.187884+0000","last_unstale":"2024-03-18T15:57:58.173475+0000","last_undegraded":"2024-03-18T15:57:58.173475+0000","last_fullsized":"2024-03-18T15:57:58.173475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:19.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173835+0000","last_change":"2024-03-18T15:57:57.182489+0000","last_active":"2024-03-18T15:57:58.173835+0000","last_peered":"2024-03-18T15:57:58.173835+0000","last_clean":"2024-03-18T15:57:58.173835+0000","last_became_active":"2024-03-18T15:57:57.176462+0000","last_became_peered":"2024-03-18T15:57:57.176462+0000","last_unstale":"2024-03-18T15:57:58.173835+0000","last_undegraded":"2024-03-18T15:57:58.173835+0000","last_fullsized":"2024-03-18T15:57:58.173835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:46.138422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173007+0000","last_change":"2024-03-18T15:57:57.185848+0000","last_active":"2024-03-18T15:57:58.173007+0000","last_peered":"2024-03-18T15:57:58.173007+0000","last_clean":"2024-03-18T15:57:58.173007+0000","last_became_active":"2024-03-18T15:57:57.185507+0000","last_became_peered":"2024-03-18T15:57:57.185507+0000","last_unstale":"2024-03-18T15:57:58.173007+0000","last_undegraded":"2024-03-18T15:57:58.173007+0000","last_fullsized":"2024-03-18T15:57:58.173007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:09.319325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:58.173733+0000","last_change":"2024-03-18T15:57:57.189222+0000","last_active":"2024-03-18T15:57:58.173733+0000","last_peered":"2024-03-18T15:57:58.173733+0000","last_clean":"2024-03-18T15:57:58.173733+0000","last_became_active":"2024-03-18T15:57:57.188847+0000","last_became_peered":"2024-03-18T15:57:57.188847+0000","last_unstale":"2024-03-18T15:57:58.173733+0000","last_undegraded":"2024-03-18T15:57:58.173733+0000","last_fullsized":"2024-03-18T15:57:58.173733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:15.935552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169947+0000","last_change":"2024-03-18T15:57:57.183269+0000","last_active":"2024-03-18T15:57:59.169947+0000","last_peered":"2024-03-18T15:57:59.169947+0000","last_clean":"2024-03-18T15:57:59.169947+0000","last_became_active":"2024-03-18T15:57:57.182866+0000","last_became_peered":"2024-03-18T15:57:57.182866+0000","last_unstale":"2024-03-18T15:57:59.169947+0000","last_undegraded":"2024-03-18T15:57:59.169947+0000","last_fullsized":"2024-03-18T15:57:59.169947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:15:08.500842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T15:57:59.169644+0000","last_change":"2024-03-18T15:57:57.179926+0000","last_active":"2024-03-18T15:57:59.169644+0000","last_peered":"2024-03-18T15:57:59.169644+0000","last_clean":"2024-03-18T15:57:59.169644+0000","last_became_active":"2024-03-18T15:57:57.179534+0000","last_became_peered":"2024-03-18T15:57:57.179534+0000","last_unstale":"2024-03-18T15:57:59.169644+0000","last_undegraded":"2024-03-18T15:57:59.169644+0000","last_fullsized":"2024-03-18T15:57:59.169644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:41.467292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285179,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5110,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":105,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":38796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":443597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":1026619,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":443799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":921600,"data_stored":435049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":1035175,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":443105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":444627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":444493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":445977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":443299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":443659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":1034859,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":26860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:58:04.254 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T15:58:04.255 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T15:58:04.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T15:58:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:04 smithi156 ceph-mon[35919]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:04 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/779490158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:58:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:04 smithi156 ceph-mon[35919]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 983 B/s rd, 4.1 KiB/s wr, 15 op/s 2024-03-18T15:58:04.518 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:04 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2329708125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:04.636 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:04 smithi191 ceph-mon[36784]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:04.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:04 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/779490158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:58:04.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:04 smithi191 ceph-mon[36784]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 983 B/s rd, 4.1 KiB/s wr, 15 op/s 2024-03-18T15:58:04.637 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:04 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2329708125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:05.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:05 smithi150 ceph-mon[31946]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:05.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:05 smithi150 ceph-mon[31946]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:05.643 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:05 smithi191 ceph-mon[36784]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:05.643 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:05 smithi191 ceph-mon[36784]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:05 smithi156 ceph-mon[35919]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:05 smithi156 ceph-mon[35919]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:58:06.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 -- 172.21.15.150:0/2951959258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad60102e40 msgr2=0x7fad601096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:06.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 --2- 172.21.15.150:0/2951959258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad60102e40 0x7fad601096f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fad5800b0d0 tx=0x7fad5800b3e0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 -- 172.21.15.150:0/2951959258 shutdown_connections 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 --2- 172.21.15.150:0/2951959258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad60102e40 0x7fad601096f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 --2- 172.21.15.150:0/2951959258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 0x7fad601016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 --2- 172.21.15.150:0/2951959258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad601024a0 0x7fad60102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 -- 172.21.15.150:0/2951959258 >> 172.21.15.150:0/2951959258 conn(0x7fad600fc860 msgr2=0x7fad600fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 -- 172.21.15.150:0/2951959258 shutdown_connections 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.283+0000 7fad68699700 1 -- 172.21.15.150:0/2951959258 wait complete. 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 Processor -- start 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 -- start start 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 0x7fad601972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 0x7fad601977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 0x7fad6019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:06.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad6010cdf0 con 0x7fad601024a0 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad6010cc70 con 0x7fad60102e40 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.284+0000 7fad68699700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fad6010cf70 con 0x7fad601012a0 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad65c34700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 0x7fad601977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 0x7fad6019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad65c34700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 0x7fad601977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55056/0 (socket says 172.21.15.150:55056) 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad65c34700 1 -- 172.21.15.150:0/1485302764 learned_addr learned my addr 172.21.15.150:0/1485302764 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66435700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 0x7fad601972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 -- 172.21.15.150:0/1485302764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 msgr2=0x7fad601972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 0x7fad601972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 -- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 msgr2=0x7fad601977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 0x7fad601977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 -- 172.21.15.150:0/1485302764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6019ef10 con 0x7fad60102e40 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad66c36700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 0x7fad6019e870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad5800d010 tx=0x7fad58007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58021070 con 0x7fad60102e40 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6019f1a0 con 0x7fad60102e40 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad5801db30 con 0x7fad60102e40 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5801bba0 con 0x7fad60102e40 2024-03-18T15:58:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.285+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6019f600 con 0x7fad60102e40 2024-03-18T15:58:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.286+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad600507e0 con 0x7fad60102e40 2024-03-18T15:58:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.287+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fad58024070 con 0x7fad60102e40 2024-03-18T15:58:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.287+0000 7fad577fe700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 0x7fad4c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.288+0000 7fad66435700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 0x7fad4c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:06.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.288+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fad580cfc60 con 0x7fad60102e40 2024-03-18T15:58:06.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.288+0000 7fad66435700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 0x7fad4c0a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fad60102300 tx=0x7fad5c007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:06.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.292+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad5809cde0 con 0x7fad60102e40 2024-03-18T15:58:06.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.569+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fad6002db70 con 0x7fad60102e40 2024-03-18T15:58:06.570 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:06 smithi150 ceph-mon[31946]: pgmap v291: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-18T15:58:06.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.573+0000 7fad577fe700 1 -- 172.21.15.150:0/1485302764 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fad5809c6e0 con 0x7fad60102e40 2024-03-18T15:58:06.574 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-18T15:58:06.575 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCOZPhltwwDIhAAa8FlixlSH10vRyg/LLFH5A== 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 msgr2=0x7fad4c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 0x7fad4c0a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fad60102300 tx=0x7fad5c007400 comp rx=0 tx=0).stop 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 msgr2=0x7fad6019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 0x7fad6019e870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad5800d010 tx=0x7fad58007a20 comp rx=0 tx=0).stop 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 shutdown_connections 2024-03-18T15:58:06.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad4c0a29d0 0x7fad4c0a4e90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad60102e40 0x7fad6019e870 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad601012a0 0x7fad601972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 --2- 172.21.15.150:0/1485302764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad601024a0 0x7fad601977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 >> 172.21.15.150:0/1485302764 conn(0x7fad600fc860 msgr2=0x7fad601056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 shutdown_connections 2024-03-18T15:58:06.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:06.576+0000 7fad68699700 1 -- 172.21.15.150:0/1485302764 wait complete. 2024-03-18T15:58:06.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:06 smithi191 ceph-mon[36784]: pgmap v291: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-18T15:58:06.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:06 smithi156 ceph-mon[35919]: pgmap v291: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 9.7 KiB/s wr, 20 op/s 2024-03-18T15:58:07.225 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:58:07.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T15:58:07.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T15:58:07.271 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T15:58:07.271 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T15:58:07.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T15:58:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:07 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1485302764' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:07 smithi150 ceph-mon[31946]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:07 smithi150 ceph-mon[31946]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:58:07.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:07 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1485302764' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:07 smithi191 ceph-mon[36784]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:07 smithi191 ceph-mon[36784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:58:07.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:07 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1485302764' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:07 smithi156 ceph-mon[35919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T15:58:07.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:07 smithi156 ceph-mon[35919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T15:58:08.546 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:08 smithi150 ceph-mon[31946]: pgmap v292: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.0 KiB/s wr, 19 op/s 2024-03-18T15:58:08.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:08 smithi191 ceph-mon[36784]: pgmap v292: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.0 KiB/s wr, 19 op/s 2024-03-18T15:58:08.770 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:08 smithi156 ceph-mon[35919]: pgmap v292: 97 pgs: 97 active+clean; 617 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.0 KiB/s wr, 19 op/s 2024-03-18T15:58:09.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 -- 172.21.15.150:0/1727658624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 msgr2=0x7f601c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 --2- 172.21.15.150:0/1727658624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c10e050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6014009990 tx=0x7f6014009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 -- 172.21.15.150:0/1727658624 shutdown_connections 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 --2- 172.21.15.150:0/1727658624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c10e050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 --2- 172.21.15.150:0/1727658624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 0x7f601c1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 --2- 172.21.15.150:0/1727658624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 -- 172.21.15.150:0/1727658624 >> 172.21.15.150:0/1727658624 conn(0x7f601c078100 msgr2=0x7f601c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 -- 172.21.15.150:0/1727658624 shutdown_connections 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.367+0000 7f602471c700 1 -- 172.21.15.150:0/1727658624 wait complete. 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 Processor -- start 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 -- start start 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:09.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 0x7f601c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f601c111460 con 0x7f601c101b70 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f601c1112e0 con 0x7f601c1011d0 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f602471c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f601c1115e0 con 0x7f601c107210 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55068/0 (socket says 172.21.15.150:55068) 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 -- 172.21.15.150:0/3547800312 learned_addr learned my addr 172.21.15.150:0/3547800312 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f60224b8700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6022cb9700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 0x7f601c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 -- 172.21.15.150:0/3547800312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 msgr2=0x7f601c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 0x7f601c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 -- 172.21.15.150:0/3547800312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 msgr2=0x7f601c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.368+0000 7f6021cb7700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f6021cb7700 1 -- 172.21.15.150:0/3547800312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c198f90 con 0x7f601c101b70 2024-03-18T15:58:09.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f60224b8700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f6021cb7700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f600c00eb30 tx=0x7f600c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c00cbd0 con 0x7f601c101b70 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c199280 con 0x7f601c101b70 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f600c004500 con 0x7f601c101b70 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c010430 con 0x7f601c101b70 2024-03-18T15:58:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.369+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c102db0 con 0x7f601c101b70 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.370+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601c1028c0 con 0x7f601c101b70 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.371+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f600c003f00 con 0x7f601c101b70 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.372+0000 7f60137fe700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 0x7f60080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.372+0000 7f60224b8700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 0x7f60080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.372+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f600c0cee70 con 0x7f601c101b70 2024-03-18T15:58:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.372+0000 7f60224b8700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 0x7f60080a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6018010bd0 tx=0x7f6018012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:09.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.375+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f600c09bff0 con 0x7f601c101b70 2024-03-18T15:58:09.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.628+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f601c062f80 con 0x7f601c101b70 2024-03-18T15:58:09.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.633+0000 7f60137fe700 1 -- 172.21.15.150:0/3547800312 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262670 (secure 0 0 0) 0x7f600c09b8f0 con 0x7f601c101b70 2024-03-18T15:58:09.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 msgr2=0x7f60080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 0x7f60080a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6018010bd0 tx=0x7f6018012040 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 msgr2=0x7f601c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f600c00eb30 tx=0x7f600c00ee40 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 shutdown_connections 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f601c107210 0x7f601c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f60080a2920 0x7f60080a4de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f601c1011d0 0x7f601c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 --2- 172.21.15.150:0/3547800312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601c101b70 0x7f601c19eb70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:09.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 >> 172.21.15.150:0/3547800312 conn(0x7f601c078100 msgr2=0x7f601c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:09.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 shutdown_connections 2024-03-18T15:58:09.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:09.636+0000 7f602471c700 1 -- 172.21.15.150:0/3547800312 wait complete. 2024-03-18T15:58:10.357 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":3095497318},{"type":"v1","addr":"172.21.15.150:6801","nonce":3095497318}]},"active_addr":"172.21.15.150:6801/3095497318","active_change":"2024-03-18T15:50:55.359993+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.150:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3143158665}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1896654966}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2222781423}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3506843134}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":4083298608}]}]} 2024-03-18T15:58:10.364 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T15:58:10.364 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T15:58:10.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:58:10.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:10 smithi150 ceph-mon[31946]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-18T15:58:10.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:10 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3547800312' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:58:10.607 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:10 smithi156 ceph-mon[35919]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-18T15:58:10.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:10 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3547800312' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:58:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:10 smithi191 ceph-mon[36784]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-03-18T15:58:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:10 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3547800312' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T15:58:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 -- 172.21.15.150:0/2519459499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec107650 msgr2=0x7fedec10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 --2- 172.21.15.150:0/2519459499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec107650 0x7fedec10f1d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fede80099c0 tx=0x7fede8009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 -- 172.21.15.150:0/2519459499 shutdown_connections 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 --2- 172.21.15.150:0/2519459499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec107650 0x7fedec10f1d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 --2- 172.21.15.150:0/2519459499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec06aa70 0x7fedec06ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 --2- 172.21.15.150:0/2519459499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 0x7fedec106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 -- 172.21.15.150:0/2519459499 >> 172.21.15.150:0/2519459499 conn(0x7fedec077220 msgr2=0x7fedec077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 -- 172.21.15.150:0/2519459499 shutdown_connections 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.318+0000 7fedf3392700 1 -- 172.21.15.150:0/2519459499 wait complete. 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 Processor -- start 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 -- start start 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 0x7fedec19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec107650 0x7fedec1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fedec1125f0 con 0x7fedec06aa70 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fedec112470 con 0x7fedec107650 2024-03-18T15:58:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf3392700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fedec112770 con 0x7fedec06b420 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55078/0 (socket says 172.21.15.150:55078) 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 -- 172.21.15.150:0/2598443927 learned_addr learned my addr 172.21.15.150:0/2598443927 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf192f700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec107650 0x7fedec1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf092d700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 0x7fedec19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 -- 172.21.15.150:0/2598443927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 msgr2=0x7fedec19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 0x7fedec19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.319+0000 7fedf112e700 1 -- 172.21.15.150:0/2598443927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec107650 msgr2=0x7fedec1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fedf112e700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec107650 0x7fedec1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fedf112e700 1 -- 172.21.15.150:0/2598443927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedec199010 con 0x7fedec06aa70 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fedf112e700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fede400ef10 tx=0x7fede400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede400c9c0 con 0x7fedec06aa70 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede4019070 con 0x7fedec06aa70 2024-03-18T15:58:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedec1992a0 con 0x7fedec06aa70 2024-03-18T15:58:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4007e60 con 0x7fedec06aa70 2024-03-18T15:58:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.320+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedec1a55c0 con 0x7fedec06aa70 2024-03-18T15:58:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.321+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedec068f90 con 0x7fedec06aa70 2024-03-18T15:58:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.322+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fede4012050 con 0x7fedec06aa70 2024-03-18T15:58:12.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.323+0000 7fede27fc700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 0x7fedd80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.323+0000 7fedf092d700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 0x7fedd80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.323+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fede40ce920 con 0x7fedec06aa70 2024-03-18T15:58:12.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.323+0000 7fedf092d700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 0x7fedd80a4e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fedec100e30 tx=0x7feddc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:12.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.327+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fede4010040 con 0x7fedec06aa70 2024-03-18T15:58:12.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.526+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fedec02db70 con 0x7fedec06aa70 2024-03-18T15:58:12.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.528+0000 7fede27fc700 1 -- 172.21.15.150:0/2598443927 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22275 (secure 0 0 0) 0x7fede409ba10 con 0x7fedec06aa70 2024-03-18T15:58:12.529 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:12.529 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:58.165838+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:58:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 msgr2=0x7fedd80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 0x7fedd80a4e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fedec100e30 tx=0x7feddc009260 comp rx=0 tx=0).stop 2024-03-18T15:58:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 msgr2=0x7fedec19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fede400ef10 tx=0x7fede400c3a0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 shutdown_connections 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fedd80a29a0 0x7fedd80a4e60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fedec107650 0x7fedec1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec06aa70 0x7fedec19e6f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 --2- 172.21.15.150:0/2598443927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fedec06b420 0x7fedec19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 >> 172.21.15.150:0/2598443927 conn(0x7fedec077220 msgr2=0x7fedec0ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 shutdown_connections 2024-03-18T15:58:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:12.530+0000 7fedf3392700 1 -- 172.21.15.150:0/2598443927 wait complete. 2024-03-18T15:58:12.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:12 smithi156 ceph-mon[35919]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-18T15:58:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:12 smithi150 ceph-mon[31946]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-18T15:58:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:12 smithi191 ceph-mon[36784]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-18T15:58:13.199 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T15:58:13.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T15:58:13.442 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:13.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:13 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2598443927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:13 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2598443927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:13 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2598443927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:14.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:14 smithi150 ceph-mon[31946]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-03-18T15:58:14.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:14 smithi156 ceph-mon[35919]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-03-18T15:58:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:14 smithi191 ceph-mon[36784]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-03-18T15:58:15.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 -- 172.21.15.150:0/2289258175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508108320 msgr2=0x7f1508108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 --2- 172.21.15.150:0/2289258175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508108320 0x7f1508108700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f14f80099c0 tx=0x7f14f8009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 -- 172.21.15.150:0/2289258175 shutdown_connections 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 --2- 172.21.15.150:0/2289258175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f150806a920 0x7f150810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 --2- 172.21.15.150:0/2289258175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508108320 0x7f1508108700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 --2- 172.21.15.150:0/2289258175 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1508069f80 0x7f150806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 -- 172.21.15.150:0/2289258175 >> 172.21.15.150:0/2289258175 conn(0x7f1508078100 msgr2=0x7f1508078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 -- 172.21.15.150:0/2289258175 shutdown_connections 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.346+0000 7f150dd2d700 1 -- 172.21.15.150:0/2289258175 wait complete. 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 Processor -- start 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 -- start start 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 0x7f15081017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1508108320 0x7f1508101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1508112560 con 0x7f1508069f80 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15081123e0 con 0x7f1508108320 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f150dd2d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f15081126e0 con 0x7f150806a920 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f15077fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f15077fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42154/0 (socket says 172.21.15.150:42154) 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f15077fe700 1 -- 172.21.15.150:0/3436242902 learned_addr learned my addr 172.21.15.150:0/3436242902 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 0x7f15081017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1507fff700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1508108320 0x7f1508101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 -- 172.21.15.150:0/3436242902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1508108320 msgr2=0x7f1508101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1508108320 0x7f1508101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 -- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 msgr2=0x7f15081012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f1506ffd700 1 -- 172.21.15.150:0/3436242902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15081a6f20 con 0x7f150806a920 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.347+0000 7f15077fe700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f1506ffd700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 0x7f15081017e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00c9c0 con 0x7f150806a920 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14fc00cb60 con 0x7f150806a920 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15081a7210 con 0x7f150806a920 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc018680 con 0x7f150806a920 2024-03-18T15:58:15.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.348+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15081a7750 con 0x7f150806a920 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.349+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1508068f90 con 0x7f150806a920 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.350+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f14fc047a90 con 0x7f150806a920 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.350+0000 7f1504ff9700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 0x7f14f00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.350+0000 7f15077fe700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 0x7f14f00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.351+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f14fc0cec00 con 0x7f150806a920 2024-03-18T15:58:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.351+0000 7f15077fe700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 0x7f14f00a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009990 tx=0x7f14f8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:15.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.354+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14fc09bbf0 con 0x7f150806a920 2024-03-18T15:58:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.555+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f150802db70 con 0x7f150806a920 2024-03-18T15:58:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.557+0000 7f1504ff9700 1 -- 172.21.15.150:0/3436242902 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22275 (secure 0 0 0) 0x7f14fc09b4f0 con 0x7f150806a920 2024-03-18T15:58:15.558 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:58:15.558 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:57:58.165838+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:58:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 msgr2=0x7f14f00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 0x7f14f00a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009990 tx=0x7f14f8005960 comp rx=0 tx=0).stop 2024-03-18T15:58:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 msgr2=0x7f15081017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 0x7f15081017e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00ed00 comp rx=0 tx=0).stop 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 shutdown_connections 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f14f00a2920 0x7f14f00a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1508108320 0x7f1508101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1508069f80 0x7f15081012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 --2- 172.21.15.150:0/3436242902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f150806a920 0x7f15081017e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 >> 172.21.15.150:0/3436242902 conn(0x7f1508078100 msgr2=0x7f15080ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 shutdown_connections 2024-03-18T15:58:15.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:15.560+0000 7f150dd2d700 1 -- 172.21.15.150:0/3436242902 wait complete. 2024-03-18T15:58:16.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T15:58:16.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T15:58:16.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T15:58:16.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T15:58:16.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T15:58:16.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T15:58:16.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T15:58:16.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T15:58:16.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T15:58:16.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T15:58:16.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T15:58:16.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T15:58:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:16 smithi156 ceph-mon[35919]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-18T15:58:16.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:16 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3436242902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:16.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:16 smithi150 ceph-mon[31946]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-18T15:58:16.844 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:16 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3436242902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:16 smithi191 ceph-mon[36784]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-18T15:58:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:16 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3436242902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:58:17.113 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:17.113 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:17.121 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:17.122 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:17.124 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:18.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:18 smithi150 ceph-mon[31946]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-18T15:58:18.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:18 smithi156 ceph-mon[35919]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-18T15:58:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:18 smithi191 ceph-mon[36784]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-18T15:58:19.674 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:19.675 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:19.677 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:19.678 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:19.692 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:20.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:20 smithi156 ceph-mon[35919]: pgmap v298: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-18T15:58:20.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:20 smithi150 ceph-mon[31946]: pgmap v298: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-18T15:58:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:20 smithi191 ceph-mon[36784]: pgmap v298: 97 pgs: 97 active+clean; 635 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-18T15:58:22.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:22 smithi156 ceph-mon[35919]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T15:58:22.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:22 smithi150 ceph-mon[31946]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T15:58:22.891 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:22 smithi191 ceph-mon[36784]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.212+0000 7fd03b7d7700 1 -- 172.21.15.150:0/263126337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 msgr2=0x7fd03410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.212+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/263126337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd03410f1c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00b0d0 tx=0x7fd02c00b3e0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 -- 172.21.15.150:0/263126337 shutdown_connections 2024-03-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/263126337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd03410f1c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/263126337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102510 0x7fd0341028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/263126337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034102ec0 0x7fd034106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 -- 172.21.15.150:0/263126337 >> 172.21.15.150:0/263126337 conn(0x7fd0340fdd80 msgr2=0x7fd0341001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 -- 172.21.15.150:0/263126337 shutdown_connections 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 -- 172.21.15.150:0/263126337 wait complete. 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 Processor -- start 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 -- start start 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034102510 0x7fd03419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102ec0 0x7fd03419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.213+0000 7fd03b7d7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd034198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd03b7d7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd034112600 con 0x7fd0341075f0 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd03b7d7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd034112480 con 0x7fd034102510 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd03b7d7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd034112780 con 0x7fd034102ec0 2024-03-18T15:58:23.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd034198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd038d72700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102ec0 0x7fd03419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd034198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54686/0 (socket says 172.21.15.150:54686) 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039573700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034102510 0x7fd03419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 -- 172.21.15.150:0/2602941588 learned_addr learned my addr 172.21.15.150:0/2602941588 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 -- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102ec0 msgr2=0x7fd03419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102ec0 0x7fd03419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 -- 172.21.15.150:0/2602941588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034102510 msgr2=0x7fd03419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd034102510 0x7fd03419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd034198fc0 con 0x7fd0341075f0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd038d72700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd034102ec0 0x7fd03419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd039d74700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd034198860 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd02c006070 tx=0x7fd02c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c003c00 con 0x7fd0341075f0 2024-03-18T15:58:23.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd02c003da0 con 0x7fd0341075f0 2024-03-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd034199250 con 0x7fd0341075f0 2024-03-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c01b880 con 0x7fd0341075f0 2024-03-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.214+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0341a5570 con 0x7fd0341075f0 2024-03-18T15:58:23.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.215+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd034062a60 con 0x7fd0341075f0 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.216+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd02c0050d0 con 0x7fd0341075f0 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd02a7fc700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0200a2920 0x7fd0200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd039573700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0200a2920 0x7fd0200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd02c0d0240 con 0x7fd0341075f0 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd02a7fc700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] conn(0x7fd0200ac100 0x7fd0200ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd02c0020b0 con 0x7fd0200ac100 2024-03-18T15:58:23.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fd02c0d0720 con 0x7fd0341075f0 2024-03-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd039573700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0200a2920 0x7fd0200a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0341999a0 tx=0x7fd030009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.217+0000 7fd038d72700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] conn(0x7fd0200ac100 0x7fd0200ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.218+0000 7fd038d72700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] conn(0x7fd0200ac100 0x7fd0200ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:23.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.219+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== osd.8 v2:172.21.15.191:6816/2388059353 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd02c0020b0 con 0x7fd0200ac100 2024-03-18T15:58:23.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.235+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 --> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd034062ee0 con 0x7fd0200ac100 2024-03-18T15:58:23.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.235+0000 7fd02a7fc700 1 -- 172.21.15.150:0/2602941588 <== osd.8 v2:172.21.15.191:6816/2388059353 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd034062ee0 con 0x7fd0200ac100 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] conn(0x7fd0200ac100 msgr2=0x7fd0200ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] conn(0x7fd0200ac100 0x7fd0200ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0200a2920 msgr2=0x7fd0200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0200a2920 0x7fd0200a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0341999a0 tx=0x7fd030009260 comp rx=0 tx=0).stop 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 msgr2=0x7fd034198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 --2- 172.21.15.150:0/2602941588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0341075f0 0x7fd034198860 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd02c006070 tx=0x7fd02c007a20 comp rx=0 tx=0).stop 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd039d74700 1 -- 172.21.15.150:0/2602941588 reap_dead start 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 shutdown_connections 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 >> 172.21.15.150:0/2602941588 conn(0x7fd0340fdd80 msgr2=0x7fd0340fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:23.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 shutdown_connections 2024-03-18T15:58:23.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:23.236+0000 7fd03b7d7700 1 -- 172.21.15.150:0/2602941588 wait complete. 2024-03-18T15:58:24.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:24 smithi150 ceph-mon[31946]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T15:58:24.920 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:24 smithi191 ceph-mon[36784]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T15:58:25.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:24 smithi156 ceph-mon[35919]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- 172.21.15.150:0/2078875996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0108300 msgr2=0x7f24f01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 --2- 172.21.15.150:0/2078875996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0108300 0x7f24f01086e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0099c0 tx=0x7f24ec009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- 172.21.15.150:0/2078875996 shutdown_connections 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 --2- 172.21.15.150:0/2078875996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102ce0 0x7f24f010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 --2- 172.21.15.150:0/2078875996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0108300 0x7f24f01086e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 --2- 172.21.15.150:0/2078875996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24f0102320 0x7f24f01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- 172.21.15.150:0/2078875996 >> 172.21.15.150:0/2078875996 conn(0x7f24f0079400 msgr2=0x7f24f0079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- 172.21.15.150:0/2078875996 shutdown_connections 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- 172.21.15.150:0/2078875996 wait complete. 2024-03-18T15:58:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 Processor -- start 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.666+0000 7f24f8419700 1 -- start start 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 0x7f24f019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0102ce0 0x7f24f019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24f0108300 0x7f24f01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01125b0 con 0x7f24f0102ce0 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0112430 con 0x7f24f0108300 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0112730 con 0x7f24f0102320 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f61b5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 0x7f24f019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f61b5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 0x7f24f019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:59234/0 (socket says 172.21.15.150:59234) 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0102ce0 0x7f24f019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f61b5700 1 -- 172.21.15.150:0/2587819562 learned_addr learned my addr 172.21.15.150:0/2587819562 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f69b6700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24f0108300 0x7f24f01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 -- 172.21.15.150:0/2587819562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 msgr2=0x7f24f019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 0x7f24f019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 -- 172.21.15.150:0/2587819562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24f0108300 msgr2=0x7f24f01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24f0108300 0x7f24f01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0198f90 con 0x7f24f0102ce0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f61b5700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24f0102320 0x7f24f019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f59b4700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0102ce0 0x7f24f019eb70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009c50 tx=0x7f24e0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e001e070 con 0x7f24f0102ce0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e0019410 con 0x7f24f0102ce0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f0199280 con 0x7f24f0102ce0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0018610 con 0x7f24f0102ce0 2024-03-18T15:58:26.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.667+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f0074500 con 0x7f24f0102ce0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.668+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f24f0062a20 con 0x7f24f0102ce0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.669+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f24e0021070 con 0x7f24f0102ce0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24e77fe700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f24dc0a2920 0x7f24dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24f61b5700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f24dc0a2920 0x7f24dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f24e00d2720 con 0x7f24f0102ce0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24e77fe700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] conn(0x7f24dc0ac100 0x7f24dc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24e0028e50 con 0x7f24dc0ac100 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f24e00d2c00 con 0x7f24f0102ce0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.670+0000 7f24f69b6700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] conn(0x7f24dc0ac100 0x7f24dc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.671+0000 7f24f61b5700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f24dc0a2920 0x7f24dc0a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009990 tx=0x7f24ec005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.671+0000 7f24f69b6700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] conn(0x7f24dc0ac100 0x7f24dc0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.671+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== osd.3 v2:172.21.15.150:6810/2684549380 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f24e0028e50 con 0x7f24dc0ac100 2024-03-18T15:58:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.688+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 --> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f24f0062ea0 con 0x7f24dc0ac100 2024-03-18T15:58:26.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.688+0000 7f24e77fe700 1 -- 172.21.15.150:0/2587819562 <== osd.3 v2:172.21.15.150:6810/2684549380 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f24f0062ea0 con 0x7f24dc0ac100 2024-03-18T15:58:26.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] conn(0x7f24dc0ac100 msgr2=0x7f24dc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] conn(0x7f24dc0ac100 0x7f24dc0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f24dc0a2920 msgr2=0x7f24dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f24dc0a2920 0x7f24dc0a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24ec009990 tx=0x7f24ec005960 comp rx=0 tx=0).stop 2024-03-18T15:58:26.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0102ce0 msgr2=0x7f24f019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 --2- 172.21.15.150:0/2587819562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24f0102ce0 0x7f24f019eb70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009c50 tx=0x7f24e0009f60 comp rx=0 tx=0).stop 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f69b6700 1 -- 172.21.15.150:0/2587819562 reap_dead start 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 shutdown_connections 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 >> 172.21.15.150:0/2587819562 conn(0x7f24f0079400 msgr2=0x7f24f0100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 shutdown_connections 2024-03-18T15:58:26.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:26.689+0000 7f24f8419700 1 -- 172.21.15.150:0/2587819562 wait complete. 2024-03-18T15:58:26.800 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:26 smithi150 ceph-mon[31946]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T15:58:26.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:26 smithi191 ceph-mon[36784]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T15:58:27.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:26 smithi156 ceph-mon[35919]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 -- 172.21.15.150:0/4227992385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 msgr2=0x7fae281086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4227992385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae281086c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fae180099c0 tx=0x7fae18009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 -- 172.21.15.150:0/4227992385 shutdown_connections 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4227992385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae2806b330 0x7fae2810f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4227992385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae281086c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4227992385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806a970 0x7fae2806adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 -- 172.21.15.150:0/4227992385 >> 172.21.15.150:0/4227992385 conn(0x7fae28077f80 msgr2=0x7fae28078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 -- 172.21.15.150:0/4227992385 shutdown_connections 2024-03-18T15:58:27.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.221+0000 7fae2d73a700 1 -- 172.21.15.150:0/4227992385 wait complete. 2024-03-18T15:58:27.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.222+0000 7fae2d73a700 1 Processor -- start 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.222+0000 7fae2d73a700 1 -- start start 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.222+0000 7fae2d73a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae2806a970 0x7fae2819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806b330 0x7fae2819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae28198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae28112590 con 0x7fae281082e0 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fae28112410 con 0x7fae2806a970 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fae28112710 con 0x7fae2806b330 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae277fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae28198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae277fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae28198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54740/0 (socket says 172.21.15.150:54740) 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae277fe700 1 -- 172.21.15.150:0/4070982523 learned_addr learned my addr 172.21.15.150:0/4070982523 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806b330 0x7fae2819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:27.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 -- 172.21.15.150:0/4070982523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae2806a970 msgr2=0x7fae2819e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae26ffd700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae2806a970 0x7fae2819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae2806a970 0x7fae2819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 -- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 msgr2=0x7fae28198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae28198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae28198fd0 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae277fe700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae281082e0 0x7fae28198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae1e7fc700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806b330 0x7fae2819ebf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae10009be0 tx=0x7fae10009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1000bb30 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae281992c0 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae10004d10 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.223+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10019410 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.224+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae28101500 con 0x7fae2806b330 2024-03-18T15:58:27.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.224+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae280629a0 con 0x7fae2806b330 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fae10020020 con 0x7fae2806b330 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae140a2920 0x7fae140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae26ffd700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae140a2920 0x7fae140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fae100d2d90 con 0x7fae2806b330 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] conn(0x7fae140ac100 0x7fae140ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae140aebd0 con 0x7fae140ac100 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fae100d3140 con 0x7fae2806b330 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.226+0000 7fae277fe700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] conn(0x7fae140ac100 0x7fae140ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.227+0000 7fae26ffd700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae140a2920 0x7fae140a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fae18006010 tx=0x7fae18005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.227+0000 7fae277fe700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] conn(0x7fae140ac100 0x7fae140ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:27.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.228+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== osd.9 v2:172.21.15.150:6826/145936256 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae140aebd0 con 0x7fae140ac100 2024-03-18T15:58:27.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.243+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 --> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fae28062e20 con 0x7fae140ac100 2024-03-18T15:58:27.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae24ff9700 1 -- 172.21.15.150:0/4070982523 <== osd.9 v2:172.21.15.150:6826/145936256 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fae28062e20 con 0x7fae140ac100 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] conn(0x7fae140ac100 msgr2=0x7fae140ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] conn(0x7fae140ac100 0x7fae140ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae140a2920 msgr2=0x7fae140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fae140a2920 0x7fae140a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fae18006010 tx=0x7fae18005e80 comp rx=0 tx=0).stop 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806b330 msgr2=0x7fae2819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 --2- 172.21.15.150:0/4070982523 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae2806b330 0x7fae2819ebf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae10009be0 tx=0x7fae10009ef0 comp rx=0 tx=0).stop 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae277fe700 1 -- 172.21.15.150:0/4070982523 reap_dead start 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 shutdown_connections 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 >> 172.21.15.150:0/4070982523 conn(0x7fae28077f80 msgr2=0x7fae28100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 shutdown_connections 2024-03-18T15:58:27.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.244+0000 7fae2d73a700 1 -- 172.21.15.150:0/4070982523 wait complete. 2024-03-18T15:58:28.068 INFO:teuthology.orchestra.run.smithi150.stdout:292057776158 2024-03-18T15:58:28.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 -- 172.21.15.150:0/730963305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 msgr2=0x7f7900101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 --2- 172.21.15.150:0/730963305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f7900101630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009990 tx=0x7f78f0009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 -- 172.21.15.150:0/730963305 shutdown_connections 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 --2- 172.21.15.150:0/730963305 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7900101b70 0x7f790010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 --2- 172.21.15.150:0/730963305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7900107210 0x7f79001075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 --2- 172.21.15.150:0/730963305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f7900101630 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 -- 172.21.15.150:0/730963305 >> 172.21.15.150:0/730963305 conn(0x7f7900078100 msgr2=0x7f7900078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 -- 172.21.15.150:0/730963305 shutdown_connections 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 -- 172.21.15.150:0/730963305 wait complete. 2024-03-18T15:58:28.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.432+0000 7f790759f700 1 Processor -- start 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 -- start start 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f79001998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7900101b70 0x7f7900199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7900107210 0x7f79001a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7900111490 con 0x7f79001011d0 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7900111310 con 0x7f7900101b70 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7900111610 con 0x7f7900107210 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f79001998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f79001998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54758/0 (socket says 172.21.15.150:54758) 2024-03-18T15:58:28.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 -- 172.21.15.150:0/1641541136 learned_addr learned my addr 172.21.15.150:0/1641541136 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f7905b3c700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7900107210 0x7f79001a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f7904b3a700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7900101b70 0x7f7900199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 -- 172.21.15.150:0/1641541136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7900107210 msgr2=0x7f79001a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7900107210 0x7f79001a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 -- 172.21.15.150:0/1641541136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7900101b70 msgr2=0x7f7900199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7900101b70 0x7f7900199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79001a4eb0 con 0x7f79001011d0 2024-03-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790533b700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f79001998f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00e9a0 tx=0x7f78fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00ca10 con 0x7f79001011d0 2024-03-18T15:58:28.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79001a50e0 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.433+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc00cbb0 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.434+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc018680 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.434+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79001a5640 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.434+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79000629a0 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.436+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f78fc010430 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.436+0000 7f78f67fc700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f78ec0a2920 0x7f78ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.436+0000 7f7904b3a700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f78ec0a2920 0x7f78ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.436+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f78fc0cf3f0 con 0x7f79001011d0 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f78f67fc700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] conn(0x7f78ec0ac100 0x7f78ec0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78fc01bec0 con 0x7f78ec0ac100 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f78fc0cf8d0 con 0x7f79001011d0 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f7905b3c700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] conn(0x7f78ec0ac100 0x7f78ec0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f7904b3a700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f78ec0a2920 0x7f78ec0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78f00098e0 tx=0x7f78f0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.437+0000 7f7905b3c700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] conn(0x7f78ec0ac100 0x7f78ec0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.438+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== osd.6 v2:172.21.15.150:6818/4061900525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78fc01bec0 con 0x7f78ec0ac100 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 --> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7900062da0 con 0x7f78ec0ac100 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f78f67fc700 1 -- 172.21.15.150:0/1641541136 <== osd.6 v2:172.21.15.150:6818/4061900525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7900062da0 con 0x7f78ec0ac100 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] conn(0x7f78ec0ac100 msgr2=0x7f78ec0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] conn(0x7f78ec0ac100 0x7f78ec0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f78ec0a2920 msgr2=0x7f78ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f78ec0a2920 0x7f78ec0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78f00098e0 tx=0x7f78f0005960 comp rx=0 tx=0).stop 2024-03-18T15:58:28.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 msgr2=0x7f79001998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 --2- 172.21.15.150:0/1641541136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79001011d0 0x7f79001998f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00e9a0 tx=0x7f78fc00ecb0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f7905b3c700 1 -- 172.21.15.150:0/1641541136 reap_dead start 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 shutdown_connections 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 >> 172.21.15.150:0/1641541136 conn(0x7f7900078100 msgr2=0x7f7900100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 shutdown_connections 2024-03-18T15:58:28.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.455+0000 7f790759f700 1 -- 172.21.15.150:0/1641541136 wait complete. 2024-03-18T15:58:28.777 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:28 smithi150 ceph-mon[31946]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-18T15:58:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:28 smithi191 ceph-mon[36784]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-18T15:58:29.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:28 smithi156 ceph-mon[35919]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-18T15:58:30.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 -- 172.21.15.150:0/602478285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 msgr2=0x7f47c4101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 --2- 172.21.15.150:0/602478285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c4101840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f47b80099c0 tx=0x7f47b8009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 -- 172.21.15.150:0/602478285 shutdown_connections 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 --2- 172.21.15.150:0/602478285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 0x7f47c410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 --2- 172.21.15.150:0/602478285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c4101840 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 --2- 172.21.15.150:0/602478285 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47c4101e10 0x7f47c4105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 -- 172.21.15.150:0/602478285 >> 172.21.15.150:0/602478285 conn(0x7f47c4077220 msgr2=0x7f47c4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 -- 172.21.15.150:0/602478285 shutdown_connections 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.066+0000 7f47cac21700 1 -- 172.21.15.150:0/602478285 wait complete. 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 Processor -- start 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 -- start start 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47c4101e10 0x7f47c419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 0x7f47c41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f47c41125b0 con 0x7f47c4101460 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47c4112430 con 0x7f47c41064b0 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47cac21700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f47c4112730 con 0x7f47c4101e10 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54786/0 (socket says 172.21.15.150:54786) 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 -- 172.21.15.150:0/2403687299 learned_addr learned my addr 172.21.15.150:0/2403687299 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c91be700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 0x7f47c41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c3fff700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47c4101e10 0x7f47c419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 -- 172.21.15.150:0/2403687299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47c4101e10 msgr2=0x7f47c419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47c4101e10 0x7f47c419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 -- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 msgr2=0x7f47c41988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 0x7f47c41988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c89bd700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c4199010 con 0x7f47c4101460 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.067+0000 7f47c91be700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47c41064b0 0x7f47c41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47c89bd700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c419e6a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006010 tx=0x7f47b8003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8003680 con 0x7f47c4101460 2024-03-18T15:58:30.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47b8003e50 con 0x7f47c4101460 2024-03-18T15:58:30.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c41992a0 con 0x7f47c4101460 2024-03-18T15:58:30.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b800fe10 con 0x7f47c4101460 2024-03-18T15:58:30.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.068+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c41a5570 con 0x7f47c4101460 2024-03-18T15:58:30.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.069+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f47c4062a60 con 0x7f47c4101460 2024-03-18T15:58:30.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.070+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f47b800f460 con 0x7f47c4101460 2024-03-18T15:58:30.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.070+0000 7f47c1ffb700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f47b40a2b40 0x7f47b40a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c3fff700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f47b40a2b40 0x7f47b40a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f47b80cf7f0 con 0x7f47c4101460 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c1ffb700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] conn(0x7f47b40ac1f0 0x7f47b40ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47b801ae50 con 0x7f47b40ac1f0 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f47b80cfcd0 con 0x7f47c4101460 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c3fff700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f47b40a2b40 0x7f47b40a5000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47c4100d60 tx=0x7f47b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c91be700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] conn(0x7f47b40ac1f0 0x7f47b40ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.071+0000 7f47c91be700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] conn(0x7f47b40ac1f0 0x7f47b40ae610 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.072+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== osd.7 v2:172.21.15.156:6816/255625842 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f47b801ae50 con 0x7f47b40ac1f0 2024-03-18T15:58:30.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.088+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 --> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f47c4062e60 con 0x7f47b40ac1f0 2024-03-18T15:58:30.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47c1ffb700 1 -- 172.21.15.150:0/2403687299 <== osd.7 v2:172.21.15.156:6816/255625842 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f47c4062e60 con 0x7f47b40ac1f0 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] conn(0x7f47b40ac1f0 msgr2=0x7f47b40ae610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] conn(0x7f47b40ac1f0 0x7f47b40ae610 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f47b40a2b40 msgr2=0x7f47b40a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f47b40a2b40 0x7f47b40a5000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47c4100d60 tx=0x7f47b0009260 comp rx=0 tx=0).stop 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 msgr2=0x7f47c419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 --2- 172.21.15.150:0/2403687299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47c4101460 0x7f47c419e6a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006010 tx=0x7f47b8003a90 comp rx=0 tx=0).stop 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47c91be700 1 -- 172.21.15.150:0/2403687299 reap_dead start 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 shutdown_connections 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 >> 172.21.15.150:0/2403687299 conn(0x7f47c4077220 msgr2=0x7f47c4105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 shutdown_connections 2024-03-18T15:58:30.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:30.089+0000 7f47cac21700 1 -- 172.21.15.150:0/2403687299 wait complete. 2024-03-18T15:58:30.445 INFO:teuthology.orchestra.run.smithi150.stdout:107374182459 2024-03-18T15:58:30.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T15:58:30.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:30 smithi150 ceph-mon[31946]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-18T15:58:30.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:30 smithi191 ceph-mon[36784]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-18T15:58:31.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:30 smithi156 ceph-mon[35919]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-18T15:58:31.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 -- 172.21.15.150:0/3722773444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 msgr2=0x7f2a00105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 --2- 172.21.15.150:0/3722773444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00105e00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009990 tx=0x7f29fc009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 -- 172.21.15.150:0/3722773444 shutdown_connections 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 --2- 172.21.15.150:0/3722773444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a001064e0 0x7f2a0010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 --2- 172.21.15.150:0/3722773444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a00101420 0x7f2a00101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 --2- 172.21.15.150:0/3722773444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00105e00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 -- 172.21.15.150:0/3722773444 >> 172.21.15.150:0/3722773444 conn(0x7f2a00078140 msgr2=0x7f2a00078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 -- 172.21.15.150:0/3722773444 shutdown_connections 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.108+0000 7f2a07790700 1 -- 172.21.15.150:0/3722773444 wait complete. 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 Processor -- start 2024-03-18T15:58:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 -- start start 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a00101420 0x7f2a001998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a001064e0 0x7f2a001a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001114a0 con 0x7f2a00101dd0 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00111320 con 0x7f2a00101420 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a07790700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00111620 con 0x7f2a001064e0 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54718/0 (socket says 172.21.15.150:54718) 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 -- 172.21.15.150:0/2621629825 learned_addr learned my addr 172.21.15.150:0/2621629825 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a0552c700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a00101420 0x7f2a001998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a05d2d700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a001064e0 0x7f2a001a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 -- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a001064e0 msgr2=0x7f2a001a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2a001064e0 0x7f2a001a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 -- 172.21.15.150:0/2621629825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a00101420 msgr2=0x7f2a001998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a00101420 0x7f2a001998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a001a4e80 con 0x7f2a00101dd0 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a0552c700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a00101420 0x7f2a001998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.109+0000 7f2a04d2b700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00199e00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc00bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc026070 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29fc003d70 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a001a5050 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00fdf0 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a001a5530 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.110+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a000629e0 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.112+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f29fc029070 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f29f67fc700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ec0a2920 0x7f29ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f29fc0cffe0 con 0x7f2a00101dd0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f2a0552c700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ec0a2920 0x7f29ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f29f67fc700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] conn(0x7f29ec0ac100 0x7f29ec0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29ec0aebd0 con 0x7f29ec0ac100 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.113+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f29fc0d03f0 con 0x7f2a00101dd0 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.114+0000 7f2a05d2d700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] conn(0x7f29ec0ac100 0x7f29ec0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.114+0000 7f2a0552c700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ec0a2920 0x7f29ec0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29f0011490 tx=0x7f29f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.114+0000 7f2a05d2d700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] conn(0x7f29ec0ac100 0x7f29ec0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.115+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== osd.5 v2:172.21.15.191:6808/3189109632 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29ec0aebd0 con 0x7f29ec0ac100 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.131+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 --> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a00062e60 con 0x7f29ec0ac100 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f29f67fc700 1 -- 172.21.15.150:0/2621629825 <== osd.5 v2:172.21.15.191:6808/3189109632 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2a00062e60 con 0x7f29ec0ac100 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] conn(0x7f29ec0ac100 msgr2=0x7f29ec0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] conn(0x7f29ec0ac100 0x7f29ec0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ec0a2920 msgr2=0x7f29ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f29ec0a2920 0x7f29ec0a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29f0011490 tx=0x7f29f0009260 comp rx=0 tx=0).stop 2024-03-18T15:58:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 msgr2=0x7f2a00199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 --2- 172.21.15.150:0/2621629825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00101dd0 0x7f2a00199e00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c00 tx=0x7f29fc00bb30 comp rx=0 tx=0).stop 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a05d2d700 1 -- 172.21.15.150:0/2621629825 reap_dead start 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 shutdown_connections 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 >> 172.21.15.150:0/2621629825 conn(0x7f2a00078140 msgr2=0x7f2a000ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 shutdown_connections 2024-03-18T15:58:31.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:27.132+0000 7f2a07790700 1 -- 172.21.15.150:0/2621629825 wait complete. 2024-03-18T15:58:32.709 INFO:teuthology.orchestra.run.smithi150.stdout:167503724594 2024-03-18T15:58:32.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T15:58:32.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:32 smithi150 ceph-mon[31946]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-18T15:58:32.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:32 smithi191 ceph-mon[36784]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-18T15:58:33.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:32 smithi156 ceph-mon[35919]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-18T15:58:33.636 INFO:teuthology.orchestra.run.smithi150.stdout:257698037794 2024-03-18T15:58:33.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T15:58:34.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:34 smithi150 ceph-mon[31946]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:34.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:34 smithi191 ceph-mon[36784]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:35.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:34 smithi156 ceph-mon[35919]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:36.244 INFO:teuthology.orchestra.run.smithi150.stdout:201863462958 2024-03-18T15:58:36.245 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- 172.21.15.150:0/3046172420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c101b70 msgr2=0x7f752c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 --2- 172.21.15.150:0/3046172420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c101b70 0x7f752c10e010 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f752400b0a0 tx=0x7f752400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- 172.21.15.150:0/3046172420 shutdown_connections 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 --2- 172.21.15.150:0/3046172420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c101b70 0x7f752c10e010 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 --2- 172.21.15.150:0/3046172420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f752c107260 0x7f752c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 --2- 172.21.15.150:0/3046172420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- 172.21.15.150:0/3046172420 >> 172.21.15.150:0/3046172420 conn(0x7f752c0781e0 msgr2=0x7f752c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- 172.21.15.150:0/3046172420 shutdown_connections 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- 172.21.15.150:0/3046172420 wait complete. 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 Processor -- start 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.744+0000 7f7533afc700 1 -- start start 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f752c101b70 0x7f752c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 0x7f752c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f752c1114f0 con 0x7f752c107260 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f752c111370 con 0x7f752c101b70 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f752c111670 con 0x7f752c1011b0 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7532099700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 0x7f752c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:36368/0 (socket says 172.21.15.150:36368) 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 -- 172.21.15.150:0/686487936 learned_addr learned my addr 172.21.15.150:0/686487936 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7532099700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 0x7f752c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47940/0 (socket says 172.21.15.150:47940) 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531097700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f752c101b70 0x7f752c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 -- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f752c101b70 msgr2=0x7f752c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f752c101b70 0x7f752c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 -- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 msgr2=0x7f752c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 0x7f752c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c198fd0 con 0x7f752c1011b0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7532099700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c107260 0x7f752c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7531898700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c19e670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f752800e9a0 tx=0x7f752800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752800ca10 con 0x7f752c1011b0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f752800cbb0 con 0x7f752c1011b0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c1992c0 con 0x7f752c1011b0 2024-03-18T15:58:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.745+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528018680 con 0x7f752c1011b0 2024-03-18T15:58:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.746+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c1a54d0 con 0x7f752c1011b0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.746+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f752c1052b0 con 0x7f752c1011b0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7528010430 con 0x7f752c1011b0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f75180a2920 0x7f75180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7531097700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f75180a2920 0x7f75180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f75280cf590 con 0x7f752c1011b0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] conn(0x7f75180ac100 0x7f75180ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f752801bec0 con 0x7f75180ac100 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.748+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f75280cf940 con 0x7f752c1011b0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.749+0000 7f7531097700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f75180a2920 0x7f75180a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f752c0787e0 tx=0x7f751c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.749+0000 7f7532099700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] conn(0x7f75180ac100 0x7f75180ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.749+0000 7f7532099700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] conn(0x7f75180ac100 0x7f75180ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.750+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== osd.4 v2:172.21.15.156:6808/1341344741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752801bec0 con 0x7f75180ac100 2024-03-18T15:58:36.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 --> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f752c1028a0 con 0x7f75180ac100 2024-03-18T15:58:36.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7522ffd700 1 -- 172.21.15.150:0/686487936 <== osd.4 v2:172.21.15.156:6808/1341344741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f752c1028a0 con 0x7f75180ac100 2024-03-18T15:58:36.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] conn(0x7f75180ac100 msgr2=0x7f75180ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] conn(0x7f75180ac100 0x7f75180ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f75180a2920 msgr2=0x7f75180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f75180a2920 0x7f75180a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f752c0787e0 tx=0x7f751c009260 comp rx=0 tx=0).stop 2024-03-18T15:58:36.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 msgr2=0x7f752c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 --2- 172.21.15.150:0/686487936 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f752c1011b0 0x7f752c19e670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f752800e9a0 tx=0x7f752800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7532099700 1 -- 172.21.15.150:0/686487936 reap_dead start 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 shutdown_connections 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.766+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 >> 172.21.15.150:0/686487936 conn(0x7f752c0781e0 msgr2=0x7f752c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.767+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 shutdown_connections 2024-03-18T15:58:36.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:36.767+0000 7f7533afc700 1 -- 172.21.15.150:0/686487936 wait complete. 2024-03-18T15:58:36.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:36 smithi150 ceph-mon[31946]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:36 smithi191 ceph-mon[36784]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:37.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:36 smithi156 ceph-mon[35919]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:37.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.901+0000 7f6dd493f700 1 -- 172.21.15.150:0/670068893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 msgr2=0x7f6dd01075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.901+0000 7f6dd493f700 1 --2- 172.21.15.150:0/670068893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd01075f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800b0a0 tx=0x7f6dc800b3b0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 -- 172.21.15.150:0/670068893 shutdown_connections 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 --2- 172.21.15.150:0/670068893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd0101b70 0x7f6dd010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 --2- 172.21.15.150:0/670068893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd01075f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 --2- 172.21.15.150:0/670068893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 -- 172.21.15.150:0/670068893 >> 172.21.15.150:0/670068893 conn(0x7f6dd0078100 msgr2=0x7f6dd0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 -- 172.21.15.150:0/670068893 shutdown_connections 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 -- 172.21.15.150:0/670068893 wait complete. 2024-03-18T15:58:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 Processor -- start 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 -- start start 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.902+0000 7f6dd493f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd0101b70 0x7f6dd0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd0111470 con 0x7f6dd0107210 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd01112f0 con 0x7f6dd01011d0 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd01115f0 con 0x7f6dd0101b70 2024-03-18T15:58:37.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dced9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dcdd9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd0101b70 0x7f6dd0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd01976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:47844/0 (socket says 172.21.15.150:47844) 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dced9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd0198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47960/0 (socket says 172.21.15.150:47960) 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 -- 172.21.15.150:0/594688904 learned_addr learned my addr 172.21.15.150:0/594688904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 -- 172.21.15.150:0/594688904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd0101b70 msgr2=0x7f6dd0197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dd0101b70 0x7f6dd0197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 -- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 msgr2=0x7f6dd0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd01a6f10 con 0x7f6dd01011d0 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dced9d700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd0107210 0x7f6dd0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dce59c700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd01976a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80062a0 tx=0x7f6dc8004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8021070 con 0x7f6dd01011d0 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd01a71a0 con 0x7f6dd01011d0 2024-03-18T15:58:37.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dc8007b80 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc80052c0 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.903+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd01a76a0 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.904+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6dd0062a60 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.905+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6dc8024070 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dbf7fe700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6db80a2920 0x7f6db80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dcdd9b700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6db80a2920 0x7f6db80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6dc800f020 con 0x7f6dd01011d0 2024-03-18T15:58:37.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dbf7fe700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] conn(0x7f6db80ac100 0x7f6db80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dc8026e50 con 0x7f6db80ac100 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f6dc80d0710 con 0x7f6dd01011d0 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dced9d700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] conn(0x7f6db80ac100 0x7f6db80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.906+0000 7f6dcdd9b700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6db80a2920 0x7f6db80a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0199800 tx=0x7f6dc4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.907+0000 7f6dced9d700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] conn(0x7f6db80ac100 0x7f6db80ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:37.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.907+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== osd.0 v2:172.21.15.150:6802/4075395622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6dc8026e50 con 0x7f6db80ac100 2024-03-18T15:58:37.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.923+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 --> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6dd0062ee0 con 0x7f6db80ac100 2024-03-18T15:58:37.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dbf7fe700 1 -- 172.21.15.150:0/594688904 <== osd.0 v2:172.21.15.150:6802/4075395622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6dd0062ee0 con 0x7f6db80ac100 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] conn(0x7f6db80ac100 msgr2=0x7f6db80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] conn(0x7f6db80ac100 0x7f6db80ae520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6db80a2920 msgr2=0x7f6db80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6db80a2920 0x7f6db80a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0199800 tx=0x7f6dc4009260 comp rx=0 tx=0).stop 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 msgr2=0x7f6dd01976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 --2- 172.21.15.150:0/594688904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dd01011d0 0x7f6dd01976a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80062a0 tx=0x7f6dc8004ce0 comp rx=0 tx=0).stop 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dced9d700 1 -- 172.21.15.150:0/594688904 reap_dead start 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 shutdown_connections 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 >> 172.21.15.150:0/594688904 conn(0x7f6dd0078100 msgr2=0x7f6dd00fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 shutdown_connections 2024-03-18T15:58:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:37.924+0000 7f6dd493f700 1 -- 172.21.15.150:0/594688904 wait complete. 2024-03-18T15:58:38.483 INFO:teuthology.orchestra.run.smithi150.stdout:223338299432 2024-03-18T15:58:38.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T15:58:38.785 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:38 smithi150 ceph-mon[31946]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:38.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 -- 172.21.15.150:0/2772106582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 msgr2=0x7fa93410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 --2- 172.21.15.150:0/2772106582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93410f120 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00b0d0 tx=0x7fa92c00b3e0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 -- 172.21.15.150:0/2772106582 shutdown_connections 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 --2- 172.21.15.150:0/2772106582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93410f120 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 --2- 172.21.15.150:0/2772106582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 0x7fa9341086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 --2- 172.21.15.150:0/2772106582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93406aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 -- 172.21.15.150:0/2772106582 >> 172.21.15.150:0/2772106582 conn(0x7fa9340781e0 msgr2=0x7fa9340785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 -- 172.21.15.150:0/2772106582 shutdown_connections 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.884+0000 7fa9391c3700 1 -- 172.21.15.150:0/2772106582 wait complete. 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 Processor -- start 2024-03-18T15:58:38.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 -- start start 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 0x7fa9341987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa934112580 con 0x7fa93406b3e0 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa934112400 con 0x7fa934108300 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa9391c3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa934112700 con 0x7fa93406aa20 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa932d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa93259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa932d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:36396/0 (socket says 172.21.15.150:36396) 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa93259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93419eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47988/0 (socket says 172.21.15.150:47988) 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa932d9d700 1 -- 172.21.15.150:0/246331547 learned_addr learned my addr 172.21.15.150:0/246331547 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.885+0000 7fa93359e700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 0x7fa9341987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 -- 172.21.15.150:0/246331547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 msgr2=0x7fa9341987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 0x7fa9341987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 -- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 msgr2=0x7fa93419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa934198fa0 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa93259c700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93406b3e0 0x7fa93419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa93359e700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa934108300 0x7fa9341987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa932d9d700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93419e640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa92400e9a0 tx=0x7fa92400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92400ca10 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92400cbb0 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa934199290 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924018680 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.886+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9341a54a0 con 0x7fa93406aa20 2024-03-18T15:58:38.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.887+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa934062a60 con 0x7fa93406aa20 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.888+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fa924010430 con 0x7fa93406aa20 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa91bfff700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa91c0a29a0 0x7fa91c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa93259c700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa91c0a29a0 0x7fa91c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa9240cf4a0 con 0x7fa93406aa20 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa91bfff700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] conn(0x7fa91c0ac180 0x7fa91c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa92401bec0 con 0x7fa91c0ac180 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fa9240cf850 con 0x7fa93406aa20 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa93259c700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa91c0a29a0 0x7fa91c0a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa928009680 tx=0x7fa928009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa93359e700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] conn(0x7fa91c0ac180 0x7fa91c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.889+0000 7fa93359e700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] conn(0x7fa91c0ac180 0x7fa91c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:38.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.890+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== osd.11 v2:172.21.15.191:6824/2841370319 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa92401bec0 con 0x7fa91c0ac180 2024-03-18T15:58:38.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.906+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 --> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa934062e60 con 0x7fa91c0ac180 2024-03-18T15:58:38.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa91bfff700 1 -- 172.21.15.150:0/246331547 <== osd.11 v2:172.21.15.191:6824/2841370319 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa934062e60 con 0x7fa91c0ac180 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] conn(0x7fa91c0ac180 msgr2=0x7fa91c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] conn(0x7fa91c0ac180 0x7fa91c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa91c0a29a0 msgr2=0x7fa91c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa91c0a29a0 0x7fa91c0a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa928009680 tx=0x7fa928009260 comp rx=0 tx=0).stop 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 msgr2=0x7fa93419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 --2- 172.21.15.150:0/246331547 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa93406aa20 0x7fa93419e640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa92400e9a0 tx=0x7fa92400ecb0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa93359e700 1 -- 172.21.15.150:0/246331547 reap_dead start 2024-03-18T15:58:38.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 shutdown_connections 2024-03-18T15:58:38.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 >> 172.21.15.150:0/246331547 conn(0x7fa9340781e0 msgr2=0x7fa9340ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:38.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 shutdown_connections 2024-03-18T15:58:38.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:38.907+0000 7fa9391c3700 1 -- 172.21.15.150:0/246331547 wait complete. 2024-03-18T15:58:38.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:38 smithi191 ceph-mon[36784]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:38 smithi156 ceph-mon[35919]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:58:39.726 INFO:teuthology.orchestra.run.smithi150.stdout:347892350998 2024-03-18T15:58:39.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T15:58:40.574 INFO:teuthology.orchestra.run.smithi150.stdout:34359738445 2024-03-18T15:58:40.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T15:58:40.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:40 smithi150 ceph-mon[31946]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:40 smithi191 ceph-mon[36784]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:41.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:40 smithi156 ceph-mon[35919]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:58:41.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 -- 172.21.15.150:0/3373546986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 msgr2=0x7f30e010f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 --2- 172.21.15.150:0/3373546986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e010f190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f30d800b0d0 tx=0x7f30d800b3e0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 -- 172.21.15.150:0/3373546986 shutdown_connections 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 --2- 172.21.15.150:0/3373546986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e010f190 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 --2- 172.21.15.150:0/3373546986 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e0108350 0x7f30e0108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 --2- 172.21.15.150:0/3373546986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e006a9d0 0x7f30e006ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 -- 172.21.15.150:0/3373546986 >> 172.21.15.150:0/3373546986 conn(0x7f30e00771f0 msgr2=0x7f30e0077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 -- 172.21.15.150:0/3373546986 shutdown_connections 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.849+0000 7f30e747a700 1 -- 172.21.15.150:0/3373546986 wait complete. 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 Processor -- start 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 -- start start 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e006a9d0 0x7f30e019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:41.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e0108350 0x7f30e0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f30e01125d0 con 0x7f30e006b370 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0112450 con 0x7f30e0108350 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e747a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0112750 con 0x7f30e006a9d0 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48004/0 (socket says 172.21.15.150:48004) 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 -- 172.21.15.150:0/3694564417 learned_addr learned my addr 172.21.15.150:0/3694564417 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e5216700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e006a9d0 0x7f30e019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e5a17700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e0108350 0x7f30e0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 -- 172.21.15.150:0/3694564417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e006a9d0 msgr2=0x7f30e019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e006a9d0 0x7f30e019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 -- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e0108350 msgr2=0x7f30e0198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e0108350 0x7f30e0198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e4a15700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e0199070 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e5a17700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30e0108350 0x7f30e0198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.850+0000 7f30e5216700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30e006a9d0 0x7f30e019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30e4a15700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e019ec00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e9a0 tx=0x7f30d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d000ca10 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e0199360 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d000cbb0 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0018680 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e01a5520 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.851+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f30e0062a60 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f30d0010430 con 0x7f30e006b370 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f30cc0a2920 0x7f30cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:41.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30e5216700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f30cc0a2920 0x7f30cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f30d00cf490 con 0x7f30e006b370 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] conn(0x7f30cc0ac100 0x7f30cc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30d001bec0 con 0x7f30cc0ac100 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.853+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f30d00cf970 con 0x7f30e006b370 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.854+0000 7f30e5216700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f30cc0a2920 0x7f30cc0a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009fd0 tx=0x7f30dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.854+0000 7f30e5a17700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] conn(0x7f30cc0ac100 0x7f30cc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.854+0000 7f30e5a17700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] conn(0x7f30cc0ac100 0x7f30cc0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:41.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.855+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== osd.1 v2:172.21.15.156:6800/737563874 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d001bec0 con 0x7f30cc0ac100 2024-03-18T15:58:41.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 --> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f30e0062e60 con 0x7f30cc0ac100 2024-03-18T15:58:41.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30d67fc700 1 -- 172.21.15.150:0/3694564417 <== osd.1 v2:172.21.15.156:6800/737563874 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f30e0062e60 con 0x7f30cc0ac100 2024-03-18T15:58:41.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] conn(0x7f30cc0ac100 msgr2=0x7f30cc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] conn(0x7f30cc0ac100 0x7f30cc0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f30cc0a2920 msgr2=0x7f30cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f30cc0a2920 0x7f30cc0a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009fd0 tx=0x7f30dc009260 comp rx=0 tx=0).stop 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.871+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 msgr2=0x7f30e019ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e747a700 1 --2- 172.21.15.150:0/3694564417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006b370 0x7f30e019ec00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e9a0 tx=0x7f30d000ecb0 comp rx=0 tx=0).stop 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e5a17700 1 -- 172.21.15.150:0/3694564417 reap_dead start 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 shutdown_connections 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 >> 172.21.15.150:0/3694564417 conn(0x7f30e00771f0 msgr2=0x7f30e00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 shutdown_connections 2024-03-18T15:58:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:41.872+0000 7f30e747a700 1 -- 172.21.15.150:0/3694564417 wait complete. 2024-03-18T15:58:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 -- 172.21.15.150:0/813779727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0101e10 msgr2=0x7f21e0105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 --2- 172.21.15.150:0/813779727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0101e10 0x7f21e0105dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f21d40099c0 tx=0x7f21d4009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 -- 172.21.15.150:0/813779727 shutdown_connections 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 --2- 172.21.15.150:0/813779727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e01064b0 0x7f21e010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 --2- 172.21.15.150:0/813779727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e0101460 0x7f21e0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 --2- 172.21.15.150:0/813779727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0101e10 0x7f21e0105dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 -- 172.21.15.150:0/813779727 >> 172.21.15.150:0/813779727 conn(0x7f21e0077220 msgr2=0x7f21e0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 -- 172.21.15.150:0/813779727 shutdown_connections 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.066+0000 7f21e673e700 1 -- 172.21.15.150:0/813779727 wait complete. 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 Processor -- start 2024-03-18T15:58:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 -- start start 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e0101460 0x7f21e01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e0101e10 0x7f21e0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e01064b0 0x7f21e0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21e0111470 con 0x7f21e01064b0 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01112f0 con 0x7f21e0101e10 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e673e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01115f0 con 0x7f21e0101460 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e4cdb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e01064b0 0x7f21e0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:42.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e0101460 0x7f21e01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e4cdb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e01064b0 0x7f21e0198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48016/0 (socket says 172.21.15.150:48016) 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21e4cdb700 1 -- 172.21.15.150:0/128883088 learned_addr learned my addr 172.21.15.150:0/128883088 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21df7fe700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e0101e10 0x7f21e0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 -- 172.21.15.150:0/128883088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e0101e10 msgr2=0x7f21e0197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e0101e10 0x7f21e0197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 -- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e01064b0 msgr2=0x7f21e0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e01064b0 0x7f21e0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.067+0000 7f21dffff700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e01a6f10 con 0x7f21e0101460 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21dffff700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e0101460 0x7f21e01976a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21d000e9f0 tx=0x7f21d000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d000cb70 con 0x7f21e0101460 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e01a7200 con 0x7f21e0101460 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21d000ee90 con 0x7f21e0101460 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0003dc0 con 0x7f21e0101460 2024-03-18T15:58:42.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e01a7760 con 0x7f21e0101460 2024-03-18T15:58:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.068+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21e0062a60 con 0x7f21e0101460 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.070+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f21d0003f60 con 0x7f21e0101460 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.070+0000 7f21dd7fa700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f21c80a2920 0x7f21c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.070+0000 7f21df7fe700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f21c80a2920 0x7f21c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f21d00ceef0 con 0x7f21e0101460 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21dd7fa700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] conn(0x7f21c80ac100 0x7f21c80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21d0012ec0 con 0x7f21c80ac100 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f21d00cf2a0 con 0x7f21e0101460 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21e4cdb700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] conn(0x7f21c80ac100 0x7f21c80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21df7fe700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f21c80a2920 0x7f21c80a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000c00 tx=0x7f21d4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.071+0000 7f21e4cdb700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] conn(0x7f21c80ac100 0x7f21c80ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:42.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.072+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== osd.2 v2:172.21.15.191:6800/1228471627 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21d0012ec0 con 0x7f21c80ac100 2024-03-18T15:58:42.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.088+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 --> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21e0062ee0 con 0x7f21c80ac100 2024-03-18T15:58:42.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.088+0000 7f21dd7fa700 1 -- 172.21.15.150:0/128883088 <== osd.2 v2:172.21.15.191:6800/1228471627 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f21e0062ee0 con 0x7f21c80ac100 2024-03-18T15:58:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.088+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] conn(0x7f21c80ac100 msgr2=0x7f21c80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.088+0000 7f21e673e700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] conn(0x7f21c80ac100 0x7f21c80ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f21c80a2920 msgr2=0x7f21c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f21c80a2920 0x7f21c80a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000c00 tx=0x7f21d4005fb0 comp rx=0 tx=0).stop 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e0101460 msgr2=0x7f21e01976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 --2- 172.21.15.150:0/128883088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21e0101460 0x7f21e01976a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21d000e9f0 tx=0x7f21d000ed00 comp rx=0 tx=0).stop 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e4cdb700 1 -- 172.21.15.150:0/128883088 reap_dead start 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 shutdown_connections 2024-03-18T15:58:42.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 >> 172.21.15.150:0/128883088 conn(0x7f21e0077220 msgr2=0x7f21e00ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:42.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 shutdown_connections 2024-03-18T15:58:42.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:42.089+0000 7f21e673e700 1 -- 172.21.15.150:0/128883088 wait complete. 2024-03-18T15:58:42.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:42 smithi150 ceph-mon[31946]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:58:42.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:42 smithi191 ceph-mon[36784]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:58:43.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:42 smithi156 ceph-mon[35919]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:58:43.301 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:43.301 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:43.335 INFO:teuthology.orchestra.run.smithi150.stdout:141733920825 2024-03-18T15:58:43.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T15:58:43.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 -- 172.21.15.150:0/3351261795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f695806b420 msgr2=0x7f6958106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 --2- 172.21.15.150:0/3351261795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f695806b420 0x7f6958106f00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f694c009990 tx=0x7f694c009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 -- 172.21.15.150:0/3351261795 shutdown_connections 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 --2- 172.21.15.150:0/3351261795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69581075e0 0x7f695810f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 --2- 172.21.15.150:0/3351261795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f695806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 --2- 172.21.15.150:0/3351261795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f695806b420 0x7f6958106f00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 -- 172.21.15.150:0/3351261795 >> 172.21.15.150:0/3351261795 conn(0x7f6958077220 msgr2=0x7f6958077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 -- 172.21.15.150:0/3351261795 shutdown_connections 2024-03-18T15:58:43.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 -- 172.21.15.150:0/3351261795 wait complete. 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.595+0000 7f695cff1700 1 Processor -- start 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 -- start start 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f69581012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695806b420 0x7f6958101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69581075e0 0x7f6958101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69581125b0 con 0x7f69581075e0 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6958112430 con 0x7f695806b420 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695cff1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6958112730 con 0x7f695806aa70 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f69581012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:43.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f69581012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:46634/0 (socket says 172.21.15.150:46634) 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f6956d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69581075e0 0x7f6958101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 -- 172.21.15.150:0/3945461027 learned_addr learned my addr 172.21.15.150:0/3945461027 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f6955d9b700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695806b420 0x7f6958101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 -- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695806b420 msgr2=0x7f6958101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695806b420 0x7f6958101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 -- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69581075e0 msgr2=0x7f6958101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69581075e0 0x7f6958101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69581a6f50 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.596+0000 7f695659c700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f69581012d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f694800e9a0 tx=0x7f694800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694800ca10 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f694800cbb0 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69581a7240 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948018680 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69581a77a0 con 0x7f695806aa70 2024-03-18T15:58:43.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.597+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69580ffbd0 con 0x7f695806aa70 2024-03-18T15:58:43.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.598+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6948047a90 con 0x7f695806aa70 2024-03-18T15:58:43.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f69477fe700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69400a29a0 0x7f69400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:43.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f6955d9b700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69400a29a0 0x7f69400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:43.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f69480ceea0 con 0x7f695806aa70 2024-03-18T15:58:43.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f69477fe700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] conn(0x7f69400ac180 0x7f69400ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69400aec50 con 0x7f69400ac180 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f69480cf250 con 0x7f695806aa70 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f6956d9d700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] conn(0x7f69400ac180 0x7f69400ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.599+0000 7f6955d9b700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69400a29a0 0x7f69400a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6958102d60 tx=0x7f694c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.600+0000 7f6956d9d700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] conn(0x7f69400ac180 0x7f69400ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.601+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== osd.10 v2:172.21.15.156:6824/4270466180 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69400aec50 con 0x7f69400ac180 2024-03-18T15:58:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.616+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 --> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f69581039e0 con 0x7f69400ac180 2024-03-18T15:58:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.616+0000 7f69477fe700 1 -- 172.21.15.150:0/3945461027 <== osd.10 v2:172.21.15.156:6824/4270466180 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f69581039e0 con 0x7f69400ac180 2024-03-18T15:58:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] conn(0x7f69400ac180 msgr2=0x7f69400ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] conn(0x7f69400ac180 0x7f69400ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69400a29a0 msgr2=0x7f69400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f69400a29a0 0x7f69400a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6958102d60 tx=0x7f694c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 msgr2=0x7f69581012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 --2- 172.21.15.150:0/3945461027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695806aa70 0x7f69581012d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f694800e9a0 tx=0x7f694800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f6956d9d700 1 -- 172.21.15.150:0/3945461027 reap_dead start 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 shutdown_connections 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 >> 172.21.15.150:0/3945461027 conn(0x7f6958077220 msgr2=0x7f69580ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 shutdown_connections 2024-03-18T15:58:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:43.617+0000 7f695cff1700 1 -- 172.21.15.150:0/3945461027 wait complete. 2024-03-18T15:58:44.000 INFO:teuthology.orchestra.run.smithi150.stdout:77309411396 2024-03-18T15:58:44.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T15:58:44.714 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:44 smithi150 ceph-mon[31946]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-18T15:58:44.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:44 smithi156 ceph-mon[35919]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-18T15:58:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:44 smithi191 ceph-mon[36784]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-18T15:58:45.361 INFO:teuthology.orchestra.run.smithi150.stdout:55834574921 2024-03-18T15:58:45.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T15:58:46.480 INFO:teuthology.orchestra.run.smithi150.stdout:313532612635 2024-03-18T15:58:46.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T15:58:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:46 smithi150 ceph-mon[31946]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:46.945 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:46.946 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:46.947 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:46.950 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:46.950 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:46 smithi191 ceph-mon[36784]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:47.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:46 smithi156 ceph-mon[35919]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- 172.21.15.150:0/3668756405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1134101280 msgr2=0x7f113410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 --2- 172.21.15.150:0/3668756405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1134101280 0x7f113410f1e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f112c00b0a0 tx=0x7f112c00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- 172.21.15.150:0/3668756405 shutdown_connections 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 --2- 172.21.15.150:0/3668756405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1134101280 0x7f113410f1e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 --2- 172.21.15.150:0/3668756405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 --2- 172.21.15.150:0/3668756405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11341008c0 0x7f1134100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- 172.21.15.150:0/3668756405 >> 172.21.15.150:0/3668756405 conn(0x7f11340777d0 msgr2=0x7f11340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- 172.21.15.150:0/3668756405 shutdown_connections 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- 172.21.15.150:0/3668756405 wait complete. 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 Processor -- start 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.067+0000 7f1138cfe700 1 -- start start 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341008c0 0x7f113419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 0x7f113419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1134112700 con 0x7f11341008c0 2024-03-18T15:58:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1134112580 con 0x7f1134101280 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1138cfe700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1134112880 con 0x7f1134107340 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f113259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341008c0 0x7f113419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1132d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1132d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:46652/0 (socket says 172.21.15.150:46652) 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1132d9d700 1 -- 172.21.15.150:0/259924937 learned_addr learned my addr 172.21.15.150:0/259924937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 0x7f113419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 -- 172.21.15.150:0/259924937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 msgr2=0x7f1134198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 -- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341008c0 msgr2=0x7f113419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341008c0 0x7f113419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 -- 172.21.15.150:0/259924937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1134199080 con 0x7f1134101280 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1132d9d700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f1131d9b700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 0x7f113419eca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f112400e9f0 tx=0x7f112400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:48.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.068+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112400c9c0 con 0x7f1134101280 2024-03-18T15:58:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.069+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1134199370 con 0x7f1134101280 2024-03-18T15:58:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.069+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f112400cb60 con 0x7f1134101280 2024-03-18T15:58:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.069+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1124018680 con 0x7f1134101280 2024-03-18T15:58:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.069+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11341a5630 con 0x7f1134101280 2024-03-18T15:58:48.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.069+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1134103be0 con 0x7f1134101280 2024-03-18T15:58:48.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.071+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1124010430 con 0x7f1134101280 2024-03-18T15:58:48.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.071+0000 7f11237fe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 0x7f111c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.071+0000 7f113259c700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 0x7f111c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.071+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f11240cebf0 con 0x7f1134101280 2024-03-18T15:58:48.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.072+0000 7f113259c700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 0x7f111c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1134199a60 tx=0x7f1128009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:48.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.076+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f112409bc60 con 0x7f1134101280 2024-03-18T15:58:48.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.281+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f11341a5860 con 0x7f1134101280 2024-03-18T15:58:48.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.282+0000 7f11237fe700 1 -- 172.21.15.150:0/259924937 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f112409b560 con 0x7f1134101280 2024-03-18T15:58:48.283 INFO:teuthology.orchestra.run.smithi150.stdout:167503724598 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 msgr2=0x7f111c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 0x7f111c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1134199a60 tx=0x7f1128009260 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 msgr2=0x7f113419eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 0x7f113419eca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f112400e9f0 tx=0x7f112400ed00 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 shutdown_connections 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1134107340 0x7f1134198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341008c0 0x7f113419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f111c0a2920 0x7f111c0a4de0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 --2- 172.21.15.150:0/259924937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1134101280 0x7f113419eca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 >> 172.21.15.150:0/259924937 conn(0x7f11340777d0 msgr2=0x7f11341051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 shutdown_connections 2024-03-18T15:58:48.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:48.284+0000 7f1138cfe700 1 -- 172.21.15.150:0/259924937 wait complete. 2024-03-18T15:58:48.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:48 smithi150 ceph-mon[31946]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:58:48.810 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/259924937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:58:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:48 smithi191 ceph-mon[36784]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:58:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:48 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/259924937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:58:48.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724594 got 167503724598 for osd.5 2024-03-18T15:58:48.973 DEBUG:teuthology.parallel:result is None 2024-03-18T15:58:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:48 smithi156 ceph-mon[35919]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:58:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:48 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/259924937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T15:58:50.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:50 smithi150 ceph-mon[31946]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:58:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:50 smithi191 ceph-mon[36784]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:58:51.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:50 smithi156 ceph-mon[35919]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:58:52.816 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:52 smithi150 ceph-mon[31946]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:52 smithi191 ceph-mon[36784]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:52 smithi156 ceph-mon[35919]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:53.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 -- 172.21.15.150:0/4017480605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 msgr2=0x7f7920101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:53.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 --2- 172.21.15.150:0/4017480605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f7920101250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f79140099c0 tx=0x7f7914009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 -- 172.21.15.150:0/4017480605 shutdown_connections 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 --2- 172.21.15.150:0/4017480605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f792010f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 --2- 172.21.15.150:0/4017480605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f7920101250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 --2- 172.21.15.150:0/4017480605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 0x7f7920106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.894+0000 7f792483e700 1 -- 172.21.15.150:0/4017480605 >> 172.21.15.150:0/4017480605 conn(0x7f7920076530 msgr2=0x7f7920076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- 172.21.15.150:0/4017480605 shutdown_connections 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- 172.21.15.150:0/4017480605 wait complete. 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 Processor -- start 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- start start 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:53.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 0x7f792019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f7920198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7920112640 con 0x7f7920100e70 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79201124c0 con 0x7f7920107690 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.895+0000 7f792483e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f79201127c0 con 0x7f7920101820 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49156/0 (socket says 172.21.15.150:49156) 2024-03-18T15:58:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 -- 172.21.15.150:0/4098428395 learned_addr learned my addr 172.21.15.150:0/4098428395 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791ed9d700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f7920198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791dd9b700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 0x7f792019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 -- 172.21.15.150:0/4098428395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 msgr2=0x7f792019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 0x7f792019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 -- 172.21.15.150:0/4098428395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 msgr2=0x7f7920198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f7920198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 -- 172.21.15.150:0/4098428395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7920199080 con 0x7f7920100e70 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791ed9d700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f7920198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f791e59c700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f791400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914003680 con 0x7f7920100e70 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79140050d0 con 0x7f7920100e70 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7920199310 con 0x7f7920100e70 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914003db0 con 0x7f7920100e70 2024-03-18T15:58:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.896+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79201a5630 con 0x7f7920100e70 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.897+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f792010a8d0 con 0x7f7920100e70 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.898+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7914005280 con 0x7f7920100e70 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.899+0000 7f790f7fe700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 0x7f79080a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.899+0000 7f791dd9b700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 0x7f79080a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.899+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f79140cf880 con 0x7f7920100e70 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.899+0000 7f791dd9b700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 0x7f79080a4f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7920106fa0 tx=0x7f7910009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:53.903+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f791405d740 con 0x7f7920100e70 2024-03-18T15:58:53.927 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:53.927 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:54.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.114+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7920199e50 con 0x7f7920100e70 2024-03-18T15:58:54.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.114+0000 7f790f7fe700 1 -- 172.21.15.150:0/4098428395 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7914013070 con 0x7f7920100e70 2024-03-18T15:58:54.116 INFO:teuthology.orchestra.run.smithi150.stdout:34359738448 2024-03-18T15:58:54.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 msgr2=0x7f79080a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 0x7f79080a4f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7920106fa0 tx=0x7f7910009260 comp rx=0 tx=0).stop 2024-03-18T15:58:54.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 msgr2=0x7f792019e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f791400f690 comp rx=0 tx=0).stop 2024-03-18T15:58:54.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 shutdown_connections 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f79080a2ac0 0x7f79080a4f80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7920107690 0x7f7920198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7920100e70 0x7f792019e760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 --2- 172.21.15.150:0/4098428395 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7920101820 0x7f792019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 >> 172.21.15.150:0/4098428395 conn(0x7f7920076530 msgr2=0x7f79200fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 shutdown_connections 2024-03-18T15:58:54.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.117+0000 7f792483e700 1 -- 172.21.15.150:0/4098428395 wait complete. 2024-03-18T15:58:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 -- 172.21.15.150:0/673309112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 msgr2=0x7ffb101021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 --2- 172.21.15.150:0/673309112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb101021a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009990 tx=0x7ffaf8009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 -- 172.21.15.150:0/673309112 shutdown_connections 2024-03-18T15:58:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 --2- 172.21.15.150:0/673309112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb101026e0 0x7ffb1010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 --2- 172.21.15.150:0/673309112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb10106210 0x7ffb101065f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 --2- 172.21.15.150:0/673309112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb101021a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 -- 172.21.15.150:0/673309112 >> 172.21.15.150:0/673309112 conn(0x7ffb10076530 msgr2=0x7ffb10076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 -- 172.21.15.150:0/673309112 shutdown_connections 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 -- 172.21.15.150:0/673309112 wait complete. 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.548+0000 7ffb15033700 1 Processor -- start 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 -- start start 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb101026e0 0x7ffb10103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb10106210 0x7ffb10103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb10111530 con 0x7ffb10101d20 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffb101113b0 con 0x7ffb10106210 2024-03-18T15:58:54.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb15033700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffb101116b0 con 0x7ffb101026e0 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49178/0 (socket says 172.21.15.150:49178) 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 -- 172.21.15.150:0/3766700052 learned_addr learned my addr 172.21.15.150:0/3766700052 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb101026e0 0x7ffb10103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0f59e700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb10106210 0x7ffb10103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 -- 172.21.15.150:0/3766700052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb101026e0 msgr2=0x7ffb10103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb101026e0 0x7ffb10103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 -- 172.21.15.150:0/3766700052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb10106210 msgr2=0x7ffb10103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb10106210 0x7ffb10103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 -- 172.21.15.150:0/3766700052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb101a6f60 con 0x7ffb10101d20 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.549+0000 7ffb0ed9d700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e9c0 tx=0x7ffb0000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.550+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0000cbd0 con 0x7ffb10101d20 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.550+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb101a7250 con 0x7ffb10101d20 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.550+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb0000ee60 con 0x7ffb10101d20 2024-03-18T15:58:54.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.550+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb00005630 con 0x7ffb10101d20 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.550+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb101a7760 con 0x7ffb10101d20 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.551+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb10068f90 con 0x7ffb10101d20 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.552+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7ffb00004750 con 0x7ffb10101d20 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.552+0000 7ffb0cd99700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 0x7ffafc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.552+0000 7ffb0659c700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 0x7ffafc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.552+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ffb000cec60 con 0x7ffb10101d20 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.553+0000 7ffb0659c700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 0x7ffafc0a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800b2e0 tx=0x7ffaf8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:54.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.557+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb0009bd50 con 0x7ffb10101d20 2024-03-18T15:58:54.711 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:54 smithi150 ceph-mon[31946]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:58:54.711 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:54 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4098428395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:58:54.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.763+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ffb101a7a20 con 0x7ffb10101d20 2024-03-18T15:58:54.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.763+0000 7ffb0cd99700 1 -- 172.21.15.150:0/3766700052 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ffb0009b650 con 0x7ffb10101d20 2024-03-18T15:58:54.765 INFO:teuthology.orchestra.run.smithi150.stdout:223338299437 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 msgr2=0x7ffafc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 0x7ffafc0a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800b2e0 tx=0x7ffaf8005960 comp rx=0 tx=0).stop 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 msgr2=0x7ffb10102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e9c0 tx=0x7ffb0000ecd0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 shutdown_connections 2024-03-18T15:58:54.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffafc0a2920 0x7ffafc0a4de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb10106210 0x7ffb10103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb10101d20 0x7ffb10102f10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 --2- 172.21.15.150:0/3766700052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb101026e0 0x7ffb10103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 >> 172.21.15.150:0/3766700052 conn(0x7ffb10076530 msgr2=0x7ffb10100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 shutdown_connections 2024-03-18T15:58:54.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:54.766+0000 7ffb15033700 1 -- 172.21.15.150:0/3766700052 wait complete. 2024-03-18T15:58:54.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:54 smithi191 ceph-mon[36784]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:58:54.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:54 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4098428395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:58:55.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:54 smithi156 ceph-mon[35919]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:58:55.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:54 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4098428395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T15:58:55.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3766700052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:58:55.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:55.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-03-18T15:58:55.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:58:55.788 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-18T15:58:55.789 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:58:55.789 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-18T15:58:55.789 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T15:58:55.789 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:55 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T15:58:55.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 -- 172.21.15.150:0/2143255572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 msgr2=0x7fb148102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/2143255572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb148102990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb138009990 tx=0x7fb138009ca0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 -- 172.21.15.150:0/2143255572 shutdown_connections 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/2143255572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1481075e0 0x7fb14810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/2143255572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb148102990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/2143255572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb148102f60 0x7fb148106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 -- 172.21.15.150:0/2143255572 >> 172.21.15.150:0/2143255572 conn(0x7fb148079360 msgr2=0x7fb148079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 -- 172.21.15.150:0/2143255572 shutdown_connections 2024-03-18T15:58:55.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.909+0000 7fb14f4cf700 1 -- 172.21.15.150:0/2143255572 wait complete. 2024-03-18T15:58:55.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 Processor -- start 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 -- start start 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb148102f60 0x7fb14819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1481075e0 0x7fb148198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1481125a0 con 0x7fb1481025b0 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb148112420 con 0x7fb148102f60 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14f4cf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb148112720 con 0x7fb1481075e0 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14da6c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1481075e0 0x7fb148198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49188/0 (socket says 172.21.15.150:49188) 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 -- 172.21.15.150:0/3111458402 learned_addr learned my addr 172.21.15.150:0/3111458402 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14ca6a700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb148102f60 0x7fb14819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 -- 172.21.15.150:0/3111458402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1481075e0 msgr2=0x7fb148198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1481075e0 0x7fb148198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 -- 172.21.15.150:0/3111458402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb148102f60 msgr2=0x7fb14819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.910+0000 7fb14d26b700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb148102f60 0x7fb14819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb14d26b700 1 -- 172.21.15.150:0/3111458402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb148198fe0 con 0x7fb1481025b0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb14d26b700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb138000c00 tx=0x7fb13800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138027070 con 0x7fb1481025b0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb138003d70 con 0x7fb1481025b0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb148199270 con 0x7fb1481025b0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13800f460 con 0x7fb1481025b0 2024-03-18T15:58:55.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.911+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1481a5590 con 0x7fb1481025b0 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.912+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb148068f90 con 0x7fb1481025b0 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.913+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb13802a070 con 0x7fb1481025b0 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.914+0000 7fb13e7fc700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 0x7fb1340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.914+0000 7fb14ca6a700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 0x7fb1340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.914+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8182+0+0 (secure 0 0 0) 0x7fb1380cfd20 con 0x7fb1481025b0 2024-03-18T15:58:55.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.914+0000 7fb14ca6a700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 0x7fb1340a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb148107060 tx=0x7fb144009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:55.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:55.917+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb138013070 con 0x7fb1481025b0 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3766700052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T15:58:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:55 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T15:58:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3766700052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T15:58:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-03-18T15:58:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-18T15:58:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-18T15:58:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-18T15:58:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-18T15:58:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-18T15:58:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:55 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T15:58:56.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.120+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb14802db70 con 0x7fb1481025b0 2024-03-18T15:58:56.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.120+0000 7fb13e7fc700 1 -- 172.21.15.150:0/3111458402 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb13809cd20 con 0x7fb1481025b0 2024-03-18T15:58:56.122 INFO:teuthology.orchestra.run.smithi150.stdout:201863462963 2024-03-18T15:58:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 msgr2=0x7fb1340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 0x7fb1340a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb148107060 tx=0x7fb144009260 comp rx=0 tx=0).stop 2024-03-18T15:58:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 msgr2=0x7fb14819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb138000c00 tx=0x7fb13800bab0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 shutdown_connections 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1481075e0 0x7fb148198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1481025b0 0x7fb14819e6c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb1340a2920 0x7fb1340a4de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 --2- 172.21.15.150:0/3111458402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb148102f60 0x7fb14819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 >> 172.21.15.150:0/3111458402 conn(0x7fb148079360 msgr2=0x7fb148101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 shutdown_connections 2024-03-18T15:58:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.123+0000 7fb14f4cf700 1 -- 172.21.15.150:0/3111458402 wait complete. 2024-03-18T15:58:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 -- 172.21.15.150:0/2293475079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7241025b0 msgr2=0x7fd724102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2293475079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7241025b0 0x7fd724102990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd7140099c0 tx=0x7fd714009cd0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 -- 172.21.15.150:0/2293475079 shutdown_connections 2024-03-18T15:58:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2293475079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd724107600 0x7fd72410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2293475079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7241025b0 0x7fd724102990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2293475079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 0x7fd724106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 -- 172.21.15.150:0/2293475079 >> 172.21.15.150:0/2293475079 conn(0x7fd724079430 msgr2=0x7fd724079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 -- 172.21.15.150:0/2293475079 shutdown_connections 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 -- 172.21.15.150:0/2293475079 wait complete. 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.656+0000 7fd7298fd700 1 Processor -- start 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 -- start start 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7241025b0 0x7fd7240745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 0x7fd724074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd724112630 con 0x7fd724107600 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241124b0 con 0x7fd7241025b0 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7298fd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd7241127b0 con 0x7fd724102f60 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49212/0 (socket says 172.21.15.150:49212) 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7227fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 0x7fd724074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 -- 172.21.15.150:0/2081409254 learned_addr learned my addr 172.21.15.150:0/2081409254 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd722ffd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7241025b0 0x7fd7240745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 -- 172.21.15.150:0/2081409254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 msgr2=0x7fd724074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 0x7fd724074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 -- 172.21.15.150:0/2081409254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7241025b0 msgr2=0x7fd7240745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7241025b0 0x7fd7240745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 -- 172.21.15.150:0/2081409254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7241a7010 con 0x7fd724107600 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.657+0000 7fd7237fe700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd71c007ee0 tx=0x7fd71c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.658+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c00dd20 con 0x7fd724107600 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.658+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7241a7300 con 0x7fd724107600 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.658+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd71c004020 con 0x7fd724107600 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.658+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c015660 con 0x7fd724107600 2024-03-18T15:58:56.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.658+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7241a77f0 con 0x7fd724107600 2024-03-18T15:58:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.659+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd724068f90 con 0x7fd724107600 2024-03-18T15:58:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.660+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd71c0041d0 con 0x7fd724107600 2024-03-18T15:58:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.660+0000 7fd7288fb700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 0x7fd70c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.660+0000 7fd722ffd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 0x7fd70c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.660+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd71c0ce570 con 0x7fd724107600 2024-03-18T15:58:56.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.661+0000 7fd722ffd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 0x7fd70c0a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd714000c00 tx=0x7fd714005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.665+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd71c010070 con 0x7fd724107600 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T15:58:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:58:56.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:56 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3111458402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:58:56.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.867+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fd72402db70 con 0x7fd724107600 2024-03-18T15:58:56.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.867+0000 7fd7288fb700 1 -- 172.21.15.150:0/2081409254 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd71c09b4e0 con 0x7fd724107600 2024-03-18T15:58:56.869 INFO:teuthology.orchestra.run.smithi150.stdout:292057776164 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 msgr2=0x7fd70c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 0x7fd70c0a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd714000c00 tx=0x7fd714005fb0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 msgr2=0x7fd724075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd71c007ee0 tx=0x7fd71c00d3b0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 shutdown_connections 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724107600 0x7fd724075050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd70c0a2920 0x7fd70c0a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7241025b0 0x7fd7240745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 --2- 172.21.15.150:0/2081409254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd724102f60 0x7fd724074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.869+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 >> 172.21.15.150:0/2081409254 conn(0x7fd724079430 msgr2=0x7fd7241003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:56.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.870+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 shutdown_connections 2024-03-18T15:58:56.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:58:56.870+0000 7fd7298fd700 1 -- 172.21.15.150:0/2081409254 wait complete. 2024-03-18T15:58:56.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:56.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-03-18T15:58:56.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:58:56.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:56 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3111458402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:58:57.007 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738448 for osd.0 2024-03-18T15:58:57.007 DEBUG:teuthology.parallel:result is None 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-18T15:58:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-18T15:58:57.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-18T15:58:57.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T15:58:57.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T15:58:57.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:56 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3111458402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T15:58:57.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462963 for osd.6 2024-03-18T15:58:57.635 DEBUG:teuthology.parallel:result is None 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2081409254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:57 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:58:57.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2081409254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:58:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:57 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T15:58:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2081409254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T15:58:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:58.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:57 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:58:58.239 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299432 got 223338299437 for osd.7 2024-03-18T15:58:58.239 DEBUG:teuthology.parallel:result is None 2024-03-18T15:58:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:58:58 smithi150 ceph-mon[31946]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:58:58.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:58:58 smithi191 ceph-mon[36784]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:58:59.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:58:58 smithi156 ceph-mon[35919]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:58:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:58:59.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776158 got 292057776164 for osd.9 2024-03-18T15:58:59.771 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:00.779 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:00 smithi150 ceph-mon[31946]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:00.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 -- 172.21.15.150:0/2525687493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d18102510 msgr2=0x7f6d181028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/2525687493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d18102510 0x7f6d181028f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c0099c0 tx=0x7f6d0c009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 -- 172.21.15.150:0/2525687493 shutdown_connections 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/2525687493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d181075f0 0x7f6d1810f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/2525687493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d18102510 0x7f6d181028f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/2525687493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d18106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 -- 172.21.15.150:0/2525687493 >> 172.21.15.150:0/2525687493 conn(0x7f6d180fdd80 msgr2=0x7f6d181001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 -- 172.21.15.150:0/2525687493 shutdown_connections 2024-03-18T15:59:00.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.805+0000 7f6d1dab5700 1 -- 172.21.15.150:0/2525687493 wait complete. 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 Processor -- start 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 -- start start 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d18102510 0x7f6d1819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d1819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 0x7f6d181988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d181125f0 con 0x7f6d181075f0 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d18112470 con 0x7f6d18102510 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.806+0000 7f6d1dab5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d18112770 con 0x7f6d18102ec0 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d17fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 0x7f6d181988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d177fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d18102510 0x7f6d1819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d1819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d17fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 0x7f6d181988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49232/0 (socket says 172.21.15.150:49232) 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d17fff700 1 -- 172.21.15.150:0/764885373 learned_addr learned my addr 172.21.15.150:0/764885373 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 -- 172.21.15.150:0/764885373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d18102510 msgr2=0x7f6d1819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d18102510 0x7f6d1819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 -- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 msgr2=0x7f6d181988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 0x7f6d181988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 -- 172.21.15.150:0/764885373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d18199010 con 0x7f6d18102ec0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d16ffd700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d1819ec30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e9c0 tx=0x7f6d0800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08004220 con 0x7f6d18102ec0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d08004d10 con 0x7f6d18102ec0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d18199300 con 0x7f6d18102ec0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08005020 con 0x7f6d18102ec0 2024-03-18T15:59:00.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.807+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d181a55c0 con 0x7f6d18102ec0 2024-03-18T15:59:00.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.808+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d18068f90 con 0x7f6d18102ec0 2024-03-18T15:59:00.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.809+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6d08004750 con 0x7f6d18102ec0 2024-03-18T15:59:00.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.810+0000 7f6d14ff9700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 0x7f6d000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.810+0000 7f6d177fe700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 0x7f6d000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.810+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6d080cea10 con 0x7f6d18102ec0 2024-03-18T15:59:00.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.810+0000 7f6d177fe700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 0x7f6d000a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000c00 tx=0x7f6d0c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:00.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:00.814+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d0809b980 con 0x7f6d18102ec0 2024-03-18T15:59:00.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:00 smithi191 ceph-mon[36784]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:01.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.013+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f6d1802db70 con 0x7f6d18102ec0 2024-03-18T15:59:01.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.013+0000 7f6d14ff9700 1 -- 172.21.15.150:0/764885373 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6d0809b280 con 0x7f6d18102ec0 2024-03-18T15:59:01.015 INFO:teuthology.orchestra.run.smithi150.stdout:257698037802 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 msgr2=0x7f6d000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 0x7f6d000a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000c00 tx=0x7f6d0c005fb0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 msgr2=0x7f6d1819ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d1819ec30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e9c0 tx=0x7f6d0800ecd0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 shutdown_connections 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d181075f0 0x7f6d181988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6d000a29a0 0x7f6d000a4e60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d18102510 0x7f6d1819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 --2- 172.21.15.150:0/764885373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d18102ec0 0x7f6d1819ec30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.015+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 >> 172.21.15.150:0/764885373 conn(0x7f6d180fdd80 msgr2=0x7f6d180ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.016+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 shutdown_connections 2024-03-18T15:59:01.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:01.016+0000 7f6d1dab5700 1 -- 172.21.15.150:0/764885373 wait complete. 2024-03-18T15:59:01.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:00 smithi156 ceph-mon[35919]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:01 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/764885373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:59:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:01 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/764885373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:59:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:01 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/764885373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T15:59:02.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037794 got 257698037802 for osd.8 2024-03-18T15:59:02.043 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- 172.21.15.150:0/2394231585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 msgr2=0x7f77ec1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- 172.21.15.150:0/2394231585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1027a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f77d80099c0 tx=0x7f77d8009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- 172.21.15.150:0/2394231585 shutdown_connections 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- 172.21.15.150:0/2394231585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec102ce0 0x7f77ec10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- 172.21.15.150:0/2394231585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec108300 0x7f77ec1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- 172.21.15.150:0/2394231585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1027a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- 172.21.15.150:0/2394231585 >> 172.21.15.150:0/2394231585 conn(0x7f77ec079400 msgr2=0x7f77ec079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- 172.21.15.150:0/2394231585 shutdown_connections 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- 172.21.15.150:0/2394231585 wait complete. 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 Processor -- start 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- start start 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec102ce0 0x7f77ec1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec108300 0x7f77ec19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec1125b0 con 0x7f77ec102320 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec112430 con 0x7f77ec102ce0 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.345+0000 7f77f2e57700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec112730 con 0x7f77ec108300 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49250/0 (socket says 172.21.15.150:49250) 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 -- 172.21.15.150:0/1845483572 learned_addr learned my addr 172.21.15.150:0/1845483572 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77ebfff700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec102ce0 0x7f77ec1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f13f4700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec108300 0x7f77ec19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 -- 172.21.15.150:0/1845483572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec108300 msgr2=0x7f77ec19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec108300 0x7f77ec19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 -- 172.21.15.150:0/1845483572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec102ce0 msgr2=0x7f77ec1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec102ce0 0x7f77ec1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 -- 172.21.15.150:0/1845483572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ec19b170 con 0x7f77ec102320 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f0bf3700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f77e000e9a0 tx=0x7f77e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e000ca10 con 0x7f77ec102320 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77e000cbb0 con 0x7f77ec102320 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0018680 con 0x7f77ec102320 2024-03-18T15:59:02.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ec19b460 con 0x7f77ec102320 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.346+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ec1a7720 con 0x7f77ec102320 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.347+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77ec068f90 con 0x7f77ec102320 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.349+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f77e0010430 con 0x7f77ec102320 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.349+0000 7f77e9ffb700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 0x7f77dc0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.350+0000 7f77ebfff700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 0x7f77dc0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.350+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f77e00cedd0 con 0x7f77ec102320 2024-03-18T15:59:02.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.350+0000 7f77ebfff700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 0x7f77dc0a4ee0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009990 tx=0x7f77d8005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.354+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77e00d2030 con 0x7f77ec102320 2024-03-18T15:59:02.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.563+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f77ec1a7950 con 0x7f77ec102320 2024-03-18T15:59:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.563+0000 7f77e9ffb700 1 -- 172.21.15.150:0/1845483572 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f77e009bce0 con 0x7f77ec102320 2024-03-18T15:59:02.565 INFO:teuthology.orchestra.run.smithi150.stdout:107374182466 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.565+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 msgr2=0x7f77dc0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.565+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 0x7f77dc0a4ee0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009990 tx=0x7f77d8005a90 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 msgr2=0x7f77ec1a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f77e000e9a0 tx=0x7f77e000ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 shutdown_connections 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77ec108300 0x7f77ec19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77ec102320 0x7f77ec1a0850 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f77dc0a2a20 0x7f77dc0a4ee0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 --2- 172.21.15.150:0/1845483572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77ec102ce0 0x7f77ec1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 >> 172.21.15.150:0/1845483572 conn(0x7f77ec079400 msgr2=0x7f77ec074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 shutdown_connections 2024-03-18T15:59:02.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.566+0000 7f77f2e57700 1 -- 172.21.15.150:0/1845483572 wait complete. 2024-03-18T15:59:02.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 -- 172.21.15.150:0/3114535747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 msgr2=0x7f517410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 --2- 172.21.15.150:0/3114535747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517410e050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f516c00b0a0 tx=0x7f516c00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 -- 172.21.15.150:0/3114535747 shutdown_connections 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 --2- 172.21.15.150:0/3114535747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517410e050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 --2- 172.21.15.150:0/3114535747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 0x7f51741075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 --2- 172.21.15.150:0/3114535747 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 0x7f5174101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 -- 172.21.15.150:0/3114535747 >> 172.21.15.150:0/3114535747 conn(0x7f5174078100 msgr2=0x7f5174078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 -- 172.21.15.150:0/3114535747 shutdown_connections 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.623+0000 7f517c07c700 1 -- 172.21.15.150:0/3114535747 wait complete. 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 Processor -- start 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 -- start start 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 0x7f517419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 0x7f51741988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5174111490 con 0x7f5174101b70 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5174111310 con 0x7f5174107210 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517c07c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5174111610 con 0x7f51741011d0 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57036/0 (socket says 172.21.15.150:57036) 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 -- 172.21.15.150:0/1240090651 learned_addr learned my addr 172.21.15.150:0/1240090651 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179e18700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 0x7f517419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f517a619700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 0x7f51741988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 -- 172.21.15.150:0/1240090651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 msgr2=0x7f517419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 0x7f517419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 -- 172.21.15.150:0/1240090651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 msgr2=0x7f51741988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 0x7f51741988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 -- 172.21.15.150:0/1240090651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5174199000 con 0x7f5174101b70 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.624+0000 7f5179617700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9c0 tx=0x7f516400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.625+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516400cbd0 con 0x7f5174101b70 2024-03-18T15:59:02.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.625+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f516400ee60 con 0x7f5174101b70 2024-03-18T15:59:02.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.625+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51741992f0 con 0x7f5174101b70 2024-03-18T15:59:02.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.625+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164005630 con 0x7f5174101b70 2024-03-18T15:59:02.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.625+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51741a55b0 con 0x7f5174101b70 2024-03-18T15:59:02.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.626+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5174068f90 con 0x7f5174101b70 2024-03-18T15:59:02.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.627+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f5164004750 con 0x7f5174101b70 2024-03-18T15:59:02.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.627+0000 7f516affd700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 0x7f51600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:02.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.627+0000 7f5179e18700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 0x7f51600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:02.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.628+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f51640cf000 con 0x7f5174101b70 2024-03-18T15:59:02.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.628+0000 7f5179e18700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 0x7f51600a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51741999e0 tx=0x7f5170009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:02.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.631+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5164053740 con 0x7f5174101b70 2024-03-18T15:59:02.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.828+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f5174199dd0 con 0x7f5174101b70 2024-03-18T15:59:02.830 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:02 smithi150 ceph-mon[31946]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:02.831 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:02 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1845483572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:59:02.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.829+0000 7f516affd700 1 -- 172.21.15.150:0/1240090651 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f5174199dd0 con 0x7f5174101b70 2024-03-18T15:59:02.831 INFO:teuthology.orchestra.run.smithi150.stdout:347892351002 2024-03-18T15:59:02.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 msgr2=0x7f51600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 0x7f51600a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f51741999e0 tx=0x7f5170009260 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 msgr2=0x7f517419ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9c0 tx=0x7f516400ecd0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 shutdown_connections 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f51600a29a0 0x7f51600a4e60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5174107210 0x7f51741988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51741011d0 0x7f517419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 --2- 172.21.15.150:0/1240090651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5174101b70 0x7f517419ec20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:02.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 >> 172.21.15.150:0/1240090651 conn(0x7f5174078100 msgr2=0x7f5174100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:02.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 shutdown_connections 2024-03-18T15:59:02.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:02.831+0000 7f517c07c700 1 -- 172.21.15.150:0/1240090651 wait complete. 2024-03-18T15:59:02.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:02 smithi191 ceph-mon[36784]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:02.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:02 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1845483572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:59:03.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:02 smithi156 ceph-mon[35919]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:03.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:02 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1845483572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T15:59:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1240090651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:59:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1240090651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:59:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1240090651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T15:59:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 -- 172.21.15.150:0/900710764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db01075f0 msgr2=0x7f2db010f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 --2- 172.21.15.150:0/900710764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db01075f0 0x7f2db010f1c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2da800b0d0 tx=0x7f2da800b3e0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 -- 172.21.15.150:0/900710764 shutdown_connections 2024-03-18T15:59:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 --2- 172.21.15.150:0/900710764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db01075f0 0x7f2db010f1c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 --2- 172.21.15.150:0/900710764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102510 0x7f2db01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 --2- 172.21.15.150:0/900710764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db0102ec0 0x7f2db0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 -- 172.21.15.150:0/900710764 >> 172.21.15.150:0/900710764 conn(0x7f2db00fdd80 msgr2=0x7f2db01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:04.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 -- 172.21.15.150:0/900710764 shutdown_connections 2024-03-18T15:59:04.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.048+0000 7f2db5964700 1 -- 172.21.15.150:0/900710764 wait complete. 2024-03-18T15:59:04.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 Processor -- start 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 -- start start 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102ec0 0x7f2db019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db01075f0 0x7f2db0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112600 con 0x7f2db0102510 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112480 con 0x7f2db0102ec0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2db5964700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112780 con 0x7f2db01075f0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57054/0 (socket says 172.21.15.150:57054) 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2daeffd700 1 -- 172.21.15.150:0/1080447571 learned_addr learned my addr 172.21.15.150:0/1080447571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2da67fc700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102ec0 0x7f2db019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.049+0000 7f2daf7fe700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db01075f0 0x7f2db0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 -- 172.21.15.150:0/1080447571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db01075f0 msgr2=0x7f2db0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db01075f0 0x7f2db0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 -- 172.21.15.150:0/1080447571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102ec0 msgr2=0x7f2db019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102ec0 0x7f2db019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 -- 172.21.15.150:0/1080447571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db0199040 con 0x7f2db0102510 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2daeffd700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9a0 tx=0x7f2da000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da000ca10 con 0x7f2db0102510 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da000cbb0 con 0x7f2db0102510 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db0199330 con 0x7f2db0102510 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0018680 con 0x7f2db0102510 2024-03-18T15:59:04.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.050+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db01a54f0 con 0x7f2db0102510 2024-03-18T15:59:04.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.051+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db01039f0 con 0x7f2db0102510 2024-03-18T15:59:04.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.052+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2da0010430 con 0x7f2db0102510 2024-03-18T15:59:04.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.053+0000 7f2db4962700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 0x7f2d9c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:04.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.053+0000 7f2da67fc700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 0x7f2d9c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:04.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.053+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f2da00cee00 con 0x7f2db0102510 2024-03-18T15:59:04.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.053+0000 7f2da67fc700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 0x7f2d9c0a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2db0101490 tx=0x7f2d98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:04.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.057+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2da009bd70 con 0x7f2db0102510 2024-03-18T15:59:04.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.255+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2db01a5720 con 0x7f2db0102510 2024-03-18T15:59:04.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.255+0000 7f2db4962700 1 -- 172.21.15.150:0/1080447571 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2da009b670 con 0x7f2db0102510 2024-03-18T15:59:04.257 INFO:teuthology.orchestra.run.smithi150.stdout:141733920830 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.257+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 msgr2=0x7f2d9c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.257+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 0x7f2d9c0a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2db0101490 tx=0x7f2d98009260 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.257+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 msgr2=0x7f2db019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.257+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9a0 tx=0x7f2da000ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 shutdown_connections 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2db01075f0 0x7f2db0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db0102510 0x7f2db019e690 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f2d9c0a2920 0x7f2d9c0a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 --2- 172.21.15.150:0/1080447571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2db0102ec0 0x7f2db019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 >> 172.21.15.150:0/1080447571 conn(0x7f2db00fdd80 msgr2=0x7f2db00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:04.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 shutdown_connections 2024-03-18T15:59:04.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:04.258+0000 7f2db5964700 1 -- 172.21.15.150:0/1080447571 wait complete. 2024-03-18T15:59:04.419 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182459 got 107374182466 for osd.3 2024-03-18T15:59:04.419 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:04.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:04 smithi191 ceph-mon[36784]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:04.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:04 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1080447571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:59:05.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:04 smithi156 ceph-mon[35919]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:05.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:04 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1080447571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:59:05.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:04 smithi150 ceph-mon[31946]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:05.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:04 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1080447571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T15:59:05.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920830 for osd.4 2024-03-18T15:59:05.170 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:05.868 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351002 for osd.11 2024-03-18T15:59:05.869 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:05.911 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:05.912 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:06.902 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:06 smithi150 ceph-mon[31946]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:06 smithi191 ceph-mon[36784]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:07.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:06 smithi156 ceph-mon[35919]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T15:59:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 -- 172.21.15.150:0/2331422506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408810ac70 msgr2=0x7f408810b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 --2- 172.21.15.150:0/2331422506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408810ac70 0x7f408810b050 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4078009990 tx=0x7f4078009ca0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 -- 172.21.15.150:0/2331422506 shutdown_connections 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 --2- 172.21.15.150:0/2331422506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4088114220 0x7f4088116610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 --2- 172.21.15.150:0/2331422506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408810ac70 0x7f408810b050 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 --2- 172.21.15.150:0/2331422506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 0x7f4088113ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 -- 172.21.15.150:0/2331422506 >> 172.21.15.150:0/2331422506 conn(0x7f408806d7f0 msgr2=0x7f408806dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 -- 172.21.15.150:0/2331422506 shutdown_connections 2024-03-18T15:59:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.006+0000 7f408cc87700 1 -- 172.21.15.150:0/2331422506 wait complete. 2024-03-18T15:59:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 Processor -- start 2024-03-18T15:59:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 -- start start 2024-03-18T15:59:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 0x7f40881a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 0x7f40881a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4088119c80 con 0x7f4088114220 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4088119b00 con 0x7f408810b590 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408cc87700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4088119e00 con 0x7f408810ac70 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f4086d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 0x7f40881a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:58034/0 (socket says 172.21.15.150:58034) 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f4086d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 0x7f40881a5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57074/0 (socket says 172.21.15.150:57074) 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 -- 172.21.15.150:0/933090553 learned_addr learned my addr 172.21.15.150:0/933090553 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f4085d9b700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 0x7f40881a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 -- 172.21.15.150:0/933090553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 msgr2=0x7f40881a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 0x7f40881a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 -- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 msgr2=0x7f40881a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 0x7f40881a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.007+0000 7f408659c700 1 -- 172.21.15.150:0/933090553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40881a6220 con 0x7f408810ac70 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f408659c700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40780095b0 tx=0x7f4078005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078026070 con 0x7f408810ac70 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40881a64b0 con 0x7f408810ac70 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40780056b0 con 0x7f408810ac70 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40780173f0 con 0x7f408810ac70 2024-03-18T15:59:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.008+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40881add90 con 0x7f408810ac70 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.009+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4088068f90 con 0x7f408810ac70 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.010+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f4078029070 con 0x7f408810ac70 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.010+0000 7f40777fe700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 0x7f40700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.010+0000 7f4085d9b700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 0x7f40700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.010+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f40780cf960 con 0x7f408810ac70 2024-03-18T15:59:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.011+0000 7f4085d9b700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 0x7f40700a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4088073b80 tx=0x7f407c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:08.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.015+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f407805d740 con 0x7f408810ac70 2024-03-18T15:59:08.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.216+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f408802db70 con 0x7f408810ac70 2024-03-18T15:59:08.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.216+0000 7f40777fe700 1 -- 172.21.15.150:0/933090553 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4078013070 con 0x7f408810ac70 2024-03-18T15:59:08.218 INFO:teuthology.orchestra.run.smithi150.stdout:313532612640 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 msgr2=0x7f40700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 0x7f40700a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4088073b80 tx=0x7f407c009260 comp rx=0 tx=0).stop 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 msgr2=0x7f40881a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40780095b0 tx=0x7f4078005590 comp rx=0 tx=0).stop 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 shutdown_connections 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4088114220 0x7f40881a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f408810ac70 0x7f40881a1210 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f40700a2920 0x7f40700a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 --2- 172.21.15.150:0/933090553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f408810b590 0x7f40881a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:08.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 >> 172.21.15.150:0/933090553 conn(0x7f408806d7f0 msgr2=0x7f4088109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:08.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 shutdown_connections 2024-03-18T15:59:08.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:08.219+0000 7f408cc87700 1 -- 172.21.15.150:0/933090553 wait complete. 2024-03-18T15:59:08.903 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:08 smithi150 ceph-mon[31946]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:59:08.904 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:08 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/933090553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:59:08.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612640 for osd.10 2024-03-18T15:59:08.930 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:08 smithi191 ceph-mon[36784]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:59:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:08 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/933090553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:59:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:08 smithi156 ceph-mon[35919]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T15:59:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:08 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/933090553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T15:59:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 -- 172.21.15.150:0/1219929106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 msgr2=0x7f672810e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1219929106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f672810e0f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f672000b0d0 tx=0x7f672000b3e0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 -- 172.21.15.150:0/1219929106 shutdown_connections 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1219929106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f672810e0f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1219929106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728107440 0x7f6728107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1219929106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728101d20 0x7f67281021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 -- 172.21.15.150:0/1219929106 >> 172.21.15.150:0/1219929106 conn(0x7f6728076490 msgr2=0x7f67280768a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 -- 172.21.15.150:0/1219929106 shutdown_connections 2024-03-18T15:59:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.104+0000 7f672e1dd700 1 -- 172.21.15.150:0/1219929106 wait complete. 2024-03-18T15:59:09.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 Processor -- start 2024-03-18T15:59:09.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 -- start start 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 0x7f6728102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f67281033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728107440 0x7f6728103920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6728111600 con 0x7f67281026e0 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6728111480 con 0x7f6728101d20 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f672e1dd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6728111780 con 0x7f6728107440 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f6726ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f67281033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f67277fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 0x7f6728102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f6726ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f67281033e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57090/0 (socket says 172.21.15.150:57090) 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f6726ffd700 1 -- 172.21.15.150:0/1466770420 learned_addr learned my addr 172.21.15.150:0/1466770420 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.105+0000 7f6727fff700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728107440 0x7f6728103920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:09.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 -- 172.21.15.150:0/1466770420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728107440 msgr2=0x7f6728103920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728107440 0x7f6728103920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 -- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 msgr2=0x7f67281033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f67281033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 -- 172.21.15.150:0/1466770420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67281a7070 con 0x7f6728101d20 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f67277fe700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 0x7f6728102ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9f0 tx=0x7f671c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c00c9c0 con 0x7f6728101d20 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c00cb60 con 0x7f6728101d20 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67281a7360 con 0x7f6728101d20 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c018680 con 0x7f6728101d20 2024-03-18T15:59:09.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.106+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a78f0 con 0x7f6728101d20 2024-03-18T15:59:09.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.107+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6728068f90 con 0x7f6728101d20 2024-03-18T15:59:09.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.108+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f671c047a90 con 0x7f6728101d20 2024-03-18T15:59:09.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.109+0000 7f6724ff9700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 0x7f67100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:09.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.109+0000 7f6726ffd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 0x7f67100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:09.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.109+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f671c0cec20 con 0x7f6728101d20 2024-03-18T15:59:09.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.109+0000 7f6726ffd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 0x7f67100a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6728076a70 tx=0x7f6718009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:09.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.112+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f671c09bb30 con 0x7f6728101d20 2024-03-18T15:59:09.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.315+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f672802db70 con 0x7f6728101d20 2024-03-18T15:59:09.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.315+0000 7f6724ff9700 1 -- 172.21.15.150:0/1466770420 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f671c09b430 con 0x7f6728101d20 2024-03-18T15:59:09.317 INFO:teuthology.orchestra.run.smithi150.stdout:77309411401 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 msgr2=0x7f67100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 0x7f67100a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6728076a70 tx=0x7f6718009260 comp rx=0 tx=0).stop 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 msgr2=0x7f6728102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 0x7f6728102ea0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9f0 tx=0x7f671c00ed00 comp rx=0 tx=0).stop 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 shutdown_connections 2024-03-18T15:59:09.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6728107440 0x7f6728103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f67100a29a0 0x7f67100a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6728101d20 0x7f6728102ea0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 --2- 172.21.15.150:0/1466770420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67281026e0 0x7f67281033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 >> 172.21.15.150:0/1466770420 conn(0x7f6728076490 msgr2=0x7f6728100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 shutdown_connections 2024-03-18T15:59:09.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:09.318+0000 7f672e1dd700 1 -- 172.21.15.150:0/1466770420 wait complete. 2024-03-18T15:59:09.918 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:09 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1466770420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:59:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:09 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1466770420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:59:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:09 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1466770420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T15:59:10.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:10 smithi150 ceph-mon[31946]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T15:59:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:10 smithi191 ceph-mon[36784]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T15:59:11.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:10 smithi156 ceph-mon[35919]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T15:59:11.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411401 for osd.2 2024-03-18T15:59:11.370 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 -- 172.21.15.150:0/2672973027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c107640 msgr2=0x7f056c10f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 --2- 172.21.15.150:0/2672973027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c107640 0x7f056c10f1c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f056400b0d0 tx=0x7f056400b3e0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 -- 172.21.15.150:0/2672973027 shutdown_connections 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 --2- 172.21.15.150:0/2672973027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c107640 0x7f056c10f1c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 --2- 172.21.15.150:0/2672973027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c06abd0 0x7f056c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 --2- 172.21.15.150:0/2672973027 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06b580 0x7f056c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 -- 172.21.15.150:0/2672973027 >> 172.21.15.150:0/2672973027 conn(0x7f056c078140 msgr2=0x7f056c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:12.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 -- 172.21.15.150:0/2672973027 shutdown_connections 2024-03-18T15:59:12.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.341+0000 7f0571653700 1 -- 172.21.15.150:0/2672973027 wait complete. 2024-03-18T15:59:12.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 Processor -- start 2024-03-18T15:59:12.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 -- start start 2024-03-18T15:59:12.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 0x7f056c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 0x7f056c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f056c112600 con 0x7f056c06b580 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f056c112480 con 0x7f056c107640 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f0571653700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f056c112780 con 0x7f056c06abd0 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 0x7f056c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 0x7f056c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 0x7f056c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:58072/0 (socket says 172.21.15.150:58072) 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:36042/0 (socket says 172.21.15.150:36042) 2024-03-18T15:59:12.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.342+0000 7f056affd700 1 -- 172.21.15.150:0/4237650331 learned_addr learned my addr 172.21.15.150:0/4237650331 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 -- 172.21.15.150:0/4237650331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 msgr2=0x7f056c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 0x7f056c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 -- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 msgr2=0x7f056c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 0x7f056c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 -- 172.21.15.150:0/4237650331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c199040 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056a7fc700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 0x7f056c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f056b7fe700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f056400b770 tx=0x7f0564007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564021020 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0564004c50 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056c1992d0 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05640052b0 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.343+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056c1a55a0 con 0x7f056c107640 2024-03-18T15:59:12.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.344+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056c1039f0 con 0x7f056c107640 2024-03-18T15:59:12.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.345+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f0564024020 con 0x7f056c107640 2024-03-18T15:59:12.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.346+0000 7f0553fff700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 0x7f05540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:12.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.346+0000 7f056affd700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 0x7f05540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:12.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.346+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f056400f020 con 0x7f056c107640 2024-03-18T15:59:12.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.346+0000 7f056affd700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 0x7f05540a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f056c107180 tx=0x7f055c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:12.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.349+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f056401de20 con 0x7f056c107640 2024-03-18T15:59:12.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.603+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f056c02db70 con 0x7f056c107640 2024-03-18T15:59:12.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.603+0000 7f0553fff700 1 -- 172.21.15.150:0/4237650331 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f056409cfc0 con 0x7f056c107640 2024-03-18T15:59:12.605 INFO:teuthology.orchestra.run.smithi150.stdout:55834574927 2024-03-18T15:59:12.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 msgr2=0x7f05540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 0x7f05540a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f056c107180 tx=0x7f055c009260 comp rx=0 tx=0).stop 2024-03-18T15:59:12.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 msgr2=0x7f056c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f056400b770 tx=0x7f0564007a20 comp rx=0 tx=0).stop 2024-03-18T15:59:12.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 shutdown_connections 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f05540a2920 0x7f05540a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f056c107640 0x7f056c1988e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f056c06abd0 0x7f056c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 --2- 172.21.15.150:0/4237650331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f056c06b580 0x7f056c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 >> 172.21.15.150:0/4237650331 conn(0x7f056c078140 msgr2=0x7f056c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 shutdown_connections 2024-03-18T15:59:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:12.606+0000 7f0571653700 1 -- 172.21.15.150:0/4237650331 wait complete. 2024-03-18T15:59:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:12 smithi191 ceph-mon[36784]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:12 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:12.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:12 smithi150 ceph-mon[31946]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:12.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:12.990 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:12 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:12 smithi156 ceph-mon[35919]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:12 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:13.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574921 got 55834574927 for osd.1 2024-03-18T15:59:13.319 DEBUG:teuthology.parallel:result is None 2024-03-18T15:59:13.319 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T15:59:13.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T15:59:13.580 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:13.861 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:13 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4237650331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:59:13.861 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:13 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:13.862 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:13 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:13.862 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:13 smithi150 ceph-mon[31946]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:13 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4237650331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:59:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:13 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:13 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:13 smithi191 ceph-mon[36784]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:13 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4237650331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T15:59:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:13 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:13 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:13 smithi156 ceph-mon[35919]: from='mgr.14150 172.21.15.150:0/852169688' entity='mgr.x' 2024-03-18T15:59:14.875 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:14 smithi150 ceph-mon[31946]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:14.919 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:14 smithi156 ceph-mon[35919]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:14 smithi191 ceph-mon[36784]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 -- 172.21.15.150:0/839463849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74241083a0 msgr2=0x7f7424108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 --2- 172.21.15.150:0/839463849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74241083a0 0x7f7424108780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f741c00b0a0 tx=0x7f741c00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 -- 172.21.15.150:0/839463849 shutdown_connections 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 --2- 172.21.15.150:0/839463849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102d10 0x7f742410f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 --2- 172.21.15.150:0/839463849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74241083a0 0x7f7424108780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 --2- 172.21.15.150:0/839463849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7424102350 0x7f74241027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 -- 172.21.15.150:0/839463849 >> 172.21.15.150:0/839463849 conn(0x7f7424079390 msgr2=0x7f74240797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 -- 172.21.15.150:0/839463849 shutdown_connections 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.203+0000 7f742b1aa700 1 -- 172.21.15.150:0/839463849 wait complete. 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 Processor -- start 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 -- start start 2024-03-18T15:59:15.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7424102d10 0x7f742419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 0x7f7424198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f74241125a0 con 0x7f7424102d10 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112420 con 0x7f7424102350 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f742b1aa700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7424112720 con 0x7f74241083a0 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f7428f46700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f7429747700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 0x7f7424198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f7423fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7424102d10 0x7f742419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f7428f46700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:60908/0 (socket says 172.21.15.150:60908) 2024-03-18T15:59:15.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.204+0000 7f7428f46700 1 -- 172.21.15.150:0/443131400 learned_addr learned my addr 172.21.15.150:0/443131400 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:15.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 -- 172.21.15.150:0/443131400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 msgr2=0x7f742419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 -- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7424102d10 msgr2=0x7f742419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7424102d10 0x7f742419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 -- 172.21.15.150:0/443131400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7424198fe0 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7428f46700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7429747700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 0x7f7424198880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f741400e970 tx=0x7f741400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414004220 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7414004d10 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74241992d0 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414010700 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.205+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7424074500 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.206+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7424068f90 con 0x7f74241083a0 2024-03-18T15:59:15.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.207+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7414004750 con 0x7f74241083a0 2024-03-18T15:59:15.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.208+0000 7f7421ffb700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 0x7f740c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.208+0000 7f7428f46700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 0x7f740c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.208+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f74140ce980 con 0x7f74241083a0 2024-03-18T15:59:15.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.208+0000 7f7428f46700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 0x7f740c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f741c009fd0 tx=0x7f741c006210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:15.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.211+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f741409b890 con 0x7f74241083a0 2024-03-18T15:59:15.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.401+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7424062710 con 0x7f740c0a2920 2024-03-18T15:59:15.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.408+0000 7f7421ffb700 1 -- 172.21.15.150:0/443131400 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258876 (secure 0 0 0) 0x7f7424062710 con 0x7f740c0a2920 2024-03-18T15:59:15.413 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 msgr2=0x7f740c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 0x7f740c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f741c009fd0 tx=0x7f741c006210 comp rx=0 tx=0).stop 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 msgr2=0x7f7424198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 0x7f7424198880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f741400e970 tx=0x7f741400ec80 comp rx=0 tx=0).stop 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 shutdown_connections 2024-03-18T15:59:15.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74241083a0 0x7f7424198880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f740c0a2920 0x7f740c0a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7424102350 0x7f742419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 --2- 172.21.15.150:0/443131400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7424102d10 0x7f742419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 >> 172.21.15.150:0/443131400 conn(0x7f7424079390 msgr2=0x7f7424101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 shutdown_connections 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:15.411+0000 7f742b1aa700 1 -- 172.21.15.150:0/443131400 wait complete. 2024-03-18T15:59:15.416 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-18T15:59:16.136 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-03-18T15:59:13.462570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":580,"num_read_kb":407,"num_write":339,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377576,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564248,"statfs":{"total":1151940427776,"available":1151553789952,"internally_reserved":0,"allocated":11993088,"data_stored":7482777,"data_compressed":108786,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570404+0000","last_change":"2024-03-18T15:57:57.181734+0000","last_active":"2024-03-18T15:58:56.570404+0000","last_peered":"2024-03-18T15:58:56.570404+0000","last_clean":"2024-03-18T15:58:56.570404+0000","last_became_active":"2024-03-18T15:57:57.181321+0000","last_became_peered":"2024-03-18T15:57:57.181321+0000","last_unstale":"2024-03-18T15:58:56.570404+0000","last_undegraded":"2024-03-18T15:58:56.570404+0000","last_fullsized":"2024-03-18T15:58:56.570404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:14.921877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820601+0000","last_change":"2024-03-18T15:57:57.175040+0000","last_active":"2024-03-18T15:58:56.820601+0000","last_peered":"2024-03-18T15:58:56.820601+0000","last_clean":"2024-03-18T15:58:56.820601+0000","last_became_active":"2024-03-18T15:57:57.174721+0000","last_became_peered":"2024-03-18T15:57:57.174721+0000","last_unstale":"2024-03-18T15:58:56.820601+0000","last_undegraded":"2024-03-18T15:58:56.820601+0000","last_fullsized":"2024-03-18T15:58:56.820601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:30.844693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820544+0000","last_change":"2024-03-18T15:57:57.183409+0000","last_active":"2024-03-18T15:58:56.820544+0000","last_peered":"2024-03-18T15:58:56.820544+0000","last_clean":"2024-03-18T15:58:56.820544+0000","last_became_active":"2024-03-18T15:57:57.183089+0000","last_became_peered":"2024-03-18T15:57:57.183089+0000","last_unstale":"2024-03-18T15:58:56.820544+0000","last_undegraded":"2024-03-18T15:58:56.820544+0000","last_fullsized":"2024-03-18T15:58:56.820544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:25.702402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768596+0000","last_change":"2024-03-18T15:57:57.183104+0000","last_active":"2024-03-18T15:58:56.768596+0000","last_peered":"2024-03-18T15:58:56.768596+0000","last_clean":"2024-03-18T15:58:56.768596+0000","last_became_active":"2024-03-18T15:57:57.182750+0000","last_became_peered":"2024-03-18T15:57:57.182750+0000","last_unstale":"2024-03-18T15:58:56.768596+0000","last_undegraded":"2024-03-18T15:58:56.768596+0000","last_fullsized":"2024-03-18T15:58:56.768596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:25.369309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574242+0000","last_change":"2024-03-18T15:57:57.195272+0000","last_active":"2024-03-18T15:58:56.574242+0000","last_peered":"2024-03-18T15:58:56.574242+0000","last_clean":"2024-03-18T15:58:56.574242+0000","last_became_active":"2024-03-18T15:57:57.192605+0000","last_became_peered":"2024-03-18T15:57:57.192605+0000","last_unstale":"2024-03-18T15:58:56.574242+0000","last_undegraded":"2024-03-18T15:58:56.574242+0000","last_fullsized":"2024-03-18T15:58:56.574242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:05.719973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820641+0000","last_change":"2024-03-18T15:57:57.177918+0000","last_active":"2024-03-18T15:58:56.820641+0000","last_peered":"2024-03-18T15:58:56.820641+0000","last_clean":"2024-03-18T15:58:56.820641+0000","last_became_active":"2024-03-18T15:57:57.177551+0000","last_became_peered":"2024-03-18T15:57:57.177551+0000","last_unstale":"2024-03-18T15:58:56.820641+0000","last_undegraded":"2024-03-18T15:58:56.820641+0000","last_fullsized":"2024-03-18T15:58:56.820641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:15.033755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569391+0000","last_change":"2024-03-18T15:57:57.179306+0000","last_active":"2024-03-18T15:58:56.569391+0000","last_peered":"2024-03-18T15:58:56.569391+0000","last_clean":"2024-03-18T15:58:56.569391+0000","last_became_active":"2024-03-18T15:57:57.178955+0000","last_became_peered":"2024-03-18T15:57:57.178955+0000","last_unstale":"2024-03-18T15:58:56.569391+0000","last_undegraded":"2024-03-18T15:58:56.569391+0000","last_fullsized":"2024-03-18T15:58:56.569391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:47.343910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.822345+0000","last_change":"2024-03-18T15:58:56.821934+0000","last_active":"2024-03-18T15:58:56.822345+0000","last_peered":"2024-03-18T15:58:56.822345+0000","last_clean":"2024-03-18T15:58:56.822345+0000","last_became_active":"2024-03-18T15:58:56.821543+0000","last_became_peered":"2024-03-18T15:58:56.821543+0000","last_unstale":"2024-03-18T15:58:56.822345+0000","last_undegraded":"2024-03-18T15:58:56.822345+0000","last_fullsized":"2024-03-18T15:58:56.822345+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:06.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572683+0000","last_change":"2024-03-18T15:57:57.190203+0000","last_active":"2024-03-18T15:58:56.572683+0000","last_peered":"2024-03-18T15:58:56.572683+0000","last_clean":"2024-03-18T15:58:56.572683+0000","last_became_active":"2024-03-18T15:57:57.189850+0000","last_became_peered":"2024-03-18T15:57:57.189850+0000","last_unstale":"2024-03-18T15:58:56.572683+0000","last_undegraded":"2024-03-18T15:58:56.572683+0000","last_fullsized":"2024-03-18T15:58:56.572683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:46.274479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.821049+0000","last_change":"2024-03-18T15:57:57.171053+0000","last_active":"2024-03-18T15:58:56.821049+0000","last_peered":"2024-03-18T15:58:56.821049+0000","last_clean":"2024-03-18T15:58:56.821049+0000","last_became_active":"2024-03-18T15:57:57.170727+0000","last_became_peered":"2024-03-18T15:57:57.170727+0000","last_unstale":"2024-03-18T15:58:56.821049+0000","last_undegraded":"2024-03-18T15:58:56.821049+0000","last_fullsized":"2024-03-18T15:58:56.821049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569691+0000","last_change":"2024-03-18T15:57:57.184219+0000","last_active":"2024-03-18T15:58:56.569691+0000","last_peered":"2024-03-18T15:58:56.569691+0000","last_clean":"2024-03-18T15:58:56.569691+0000","last_became_active":"2024-03-18T15:57:57.183575+0000","last_became_peered":"2024-03-18T15:57:57.183575+0000","last_unstale":"2024-03-18T15:58:56.569691+0000","last_undegraded":"2024-03-18T15:58:56.569691+0000","last_fullsized":"2024-03-18T15:58:56.569691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:59.171634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768560+0000","last_change":"2024-03-18T15:57:57.185462+0000","last_active":"2024-03-18T15:58:56.768560+0000","last_peered":"2024-03-18T15:58:56.768560+0000","last_clean":"2024-03-18T15:58:56.768560+0000","last_became_active":"2024-03-18T15:57:57.185136+0000","last_became_peered":"2024-03-18T15:57:57.185136+0000","last_unstale":"2024-03-18T15:58:56.768560+0000","last_undegraded":"2024-03-18T15:58:56.768560+0000","last_fullsized":"2024-03-18T15:58:56.768560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:36.579297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581265+0000","last_change":"2024-03-18T15:57:57.176638+0000","last_active":"2024-03-18T15:58:56.581265+0000","last_peered":"2024-03-18T15:58:56.581265+0000","last_clean":"2024-03-18T15:58:56.581265+0000","last_became_active":"2024-03-18T15:57:57.176327+0000","last_became_peered":"2024-03-18T15:57:57.176327+0000","last_unstale":"2024-03-18T15:58:56.581265+0000","last_undegraded":"2024-03-18T15:58:56.581265+0000","last_fullsized":"2024-03-18T15:58:56.581265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:18.115306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820353+0000","last_change":"2024-03-18T15:57:57.172258+0000","last_active":"2024-03-18T15:58:56.820353+0000","last_peered":"2024-03-18T15:58:56.820353+0000","last_clean":"2024-03-18T15:58:56.820353+0000","last_became_active":"2024-03-18T15:57:57.171916+0000","last_became_peered":"2024-03-18T15:57:57.171916+0000","last_unstale":"2024-03-18T15:58:56.820353+0000","last_undegraded":"2024-03-18T15:58:56.820353+0000","last_fullsized":"2024-03-18T15:58:56.820353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:24.467000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820298+0000","last_change":"2024-03-18T15:57:57.173523+0000","last_active":"2024-03-18T15:58:56.820298+0000","last_peered":"2024-03-18T15:58:56.820298+0000","last_clean":"2024-03-18T15:58:56.820298+0000","last_became_active":"2024-03-18T15:57:57.173203+0000","last_became_peered":"2024-03-18T15:57:57.173203+0000","last_unstale":"2024-03-18T15:58:56.820298+0000","last_undegraded":"2024-03-18T15:58:56.820298+0000","last_fullsized":"2024-03-18T15:58:56.820298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:31.383153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572567+0000","last_change":"2024-03-18T15:57:57.185214+0000","last_active":"2024-03-18T15:58:56.572567+0000","last_peered":"2024-03-18T15:58:56.572567+0000","last_clean":"2024-03-18T15:58:56.572567+0000","last_became_active":"2024-03-18T15:57:57.183794+0000","last_became_peered":"2024-03-18T15:57:57.183794+0000","last_unstale":"2024-03-18T15:58:56.572567+0000","last_undegraded":"2024-03-18T15:58:56.572567+0000","last_fullsized":"2024-03-18T15:58:56.572567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:45.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570494+0000","last_change":"2024-03-18T15:57:57.183349+0000","last_active":"2024-03-18T15:58:56.570494+0000","last_peered":"2024-03-18T15:58:56.570494+0000","last_clean":"2024-03-18T15:58:56.570494+0000","last_became_active":"2024-03-18T15:57:57.182127+0000","last_became_peered":"2024-03-18T15:57:57.182127+0000","last_unstale":"2024-03-18T15:58:56.570494+0000","last_undegraded":"2024-03-18T15:58:56.570494+0000","last_fullsized":"2024-03-18T15:58:56.570494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:17.229945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568950+0000","last_change":"2024-03-18T15:57:57.182112+0000","last_active":"2024-03-18T15:58:56.568950+0000","last_peered":"2024-03-18T15:58:56.568950+0000","last_clean":"2024-03-18T15:58:56.568950+0000","last_became_active":"2024-03-18T15:57:57.181695+0000","last_became_peered":"2024-03-18T15:57:57.181695+0000","last_unstale":"2024-03-18T15:58:56.568950+0000","last_undegraded":"2024-03-18T15:58:56.568950+0000","last_fullsized":"2024-03-18T15:58:56.568950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:20.402597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572356+0000","last_change":"2024-03-18T15:57:57.182187+0000","last_active":"2024-03-18T15:58:56.572356+0000","last_peered":"2024-03-18T15:58:56.572356+0000","last_clean":"2024-03-18T15:58:56.572356+0000","last_became_active":"2024-03-18T15:57:57.181837+0000","last_became_peered":"2024-03-18T15:57:57.181837+0000","last_unstale":"2024-03-18T15:58:56.572356+0000","last_undegraded":"2024-03-18T15:58:56.572356+0000","last_fullsized":"2024-03-18T15:58:56.572356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:37:32.225077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568952+0000","last_change":"2024-03-18T15:57:56.171691+0000","last_active":"2024-03-18T15:58:56.568952+0000","last_peered":"2024-03-18T15:58:56.568952+0000","last_clean":"2024-03-18T15:58:56.568952+0000","last_became_active":"2024-03-18T15:57:56.171241+0000","last_became_peered":"2024-03-18T15:57:56.171241+0000","last_unstale":"2024-03-18T15:58:56.568952+0000","last_undegraded":"2024-03-18T15:58:56.568952+0000","last_fullsized":"2024-03-18T15:58:56.568952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:50.411828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038413+0000","last_change":"2024-03-18T15:57:57.191757+0000","last_active":"2024-03-18T15:58:57.038413+0000","last_peered":"2024-03-18T15:58:57.038413+0000","last_clean":"2024-03-18T15:58:57.038413+0000","last_became_active":"2024-03-18T15:57:57.191474+0000","last_became_peered":"2024-03-18T15:57:57.191474+0000","last_unstale":"2024-03-18T15:58:57.038413+0000","last_undegraded":"2024-03-18T15:58:57.038413+0000","last_fullsized":"2024-03-18T15:58:57.038413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:18.430632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038586+0000","last_change":"2024-03-18T15:57:56.176784+0000","last_active":"2024-03-18T15:58:57.038586+0000","last_peered":"2024-03-18T15:58:57.038586+0000","last_clean":"2024-03-18T15:58:57.038586+0000","last_became_active":"2024-03-18T15:57:56.176468+0000","last_became_peered":"2024-03-18T15:57:56.176468+0000","last_unstale":"2024-03-18T15:58:57.038586+0000","last_undegraded":"2024-03-18T15:58:57.038586+0000","last_fullsized":"2024-03-18T15:58:57.038586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:50.941724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820719+0000","last_change":"2024-03-18T15:57:57.183492+0000","last_active":"2024-03-18T15:58:56.820719+0000","last_peered":"2024-03-18T15:58:56.820719+0000","last_clean":"2024-03-18T15:58:56.820719+0000","last_became_active":"2024-03-18T15:57:57.183156+0000","last_became_peered":"2024-03-18T15:57:57.183156+0000","last_unstale":"2024-03-18T15:58:56.820719+0000","last_undegraded":"2024-03-18T15:58:56.820719+0000","last_fullsized":"2024-03-18T15:58:56.820719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:41.441788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.582136+0000","last_change":"2024-03-18T15:58:56.581740+0000","last_active":"2024-03-18T15:58:56.582136+0000","last_peered":"2024-03-18T15:58:56.582136+0000","last_clean":"2024-03-18T15:58:56.582136+0000","last_became_active":"2024-03-18T15:58:56.573608+0000","last_became_peered":"2024-03-18T15:58:56.573608+0000","last_unstale":"2024-03-18T15:58:56.582136+0000","last_undegraded":"2024-03-18T15:58:56.582136+0000","last_fullsized":"2024-03-18T15:58:56.582136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:59.170341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572372+0000","last_change":"2024-03-18T15:57:57.180588+0000","last_active":"2024-03-18T15:58:56.572372+0000","last_peered":"2024-03-18T15:58:56.572372+0000","last_clean":"2024-03-18T15:58:56.572372+0000","last_became_active":"2024-03-18T15:57:57.180245+0000","last_became_peered":"2024-03-18T15:57:57.180245+0000","last_unstale":"2024-03-18T15:58:56.572372+0000","last_undegraded":"2024-03-18T15:58:56.572372+0000","last_fullsized":"2024-03-18T15:58:56.572372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:38.095406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.575197+0000","last_change":"2024-03-18T15:57:56.179815+0000","last_active":"2024-03-18T15:58:56.575197+0000","last_peered":"2024-03-18T15:58:56.575197+0000","last_clean":"2024-03-18T15:58:56.575197+0000","last_became_active":"2024-03-18T15:57:56.179495+0000","last_became_peered":"2024-03-18T15:57:56.179495+0000","last_unstale":"2024-03-18T15:58:56.575197+0000","last_undegraded":"2024-03-18T15:58:56.575197+0000","last_fullsized":"2024-03-18T15:58:56.575197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:17.876766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573567+0000","last_change":"2024-03-18T15:57:57.180021+0000","last_active":"2024-03-18T15:58:56.573567+0000","last_peered":"2024-03-18T15:58:56.573567+0000","last_clean":"2024-03-18T15:58:56.573567+0000","last_became_active":"2024-03-18T15:57:57.179642+0000","last_became_peered":"2024-03-18T15:57:57.179642+0000","last_unstale":"2024-03-18T15:58:56.573567+0000","last_undegraded":"2024-03-18T15:58:56.573567+0000","last_fullsized":"2024-03-18T15:58:56.573567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:36.573584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:09.272376+0000","last_change":"2024-03-18T15:57:56.178907+0000","last_active":"2024-03-18T15:59:09.272376+0000","last_peered":"2024-03-18T15:59:09.272376+0000","last_clean":"2024-03-18T15:59:09.272376+0000","last_became_active":"2024-03-18T15:57:56.178420+0000","last_became_peered":"2024-03-18T15:57:56.178420+0000","last_unstale":"2024-03-18T15:59:09.272376+0000","last_undegraded":"2024-03-18T15:59:09.272376+0000","last_fullsized":"2024-03-18T15:59:09.272376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:24:45.513178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574129+0000","last_change":"2024-03-18T15:57:57.187726+0000","last_active":"2024-03-18T15:58:56.574129+0000","last_peered":"2024-03-18T15:58:56.574129+0000","last_clean":"2024-03-18T15:58:56.574129+0000","last_became_active":"2024-03-18T15:57:57.187383+0000","last_became_peered":"2024-03-18T15:57:57.187383+0000","last_unstale":"2024-03-18T15:58:56.574129+0000","last_undegraded":"2024-03-18T15:58:56.574129+0000","last_fullsized":"2024-03-18T15:58:56.574129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:23.134839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820636+0000","last_change":"2024-03-18T15:57:56.175120+0000","last_active":"2024-03-18T15:58:56.820636+0000","last_peered":"2024-03-18T15:58:56.820636+0000","last_clean":"2024-03-18T15:58:56.820636+0000","last_became_active":"2024-03-18T15:57:56.174554+0000","last_became_peered":"2024-03-18T15:57:56.174554+0000","last_unstale":"2024-03-18T15:58:56.820636+0000","last_undegraded":"2024-03-18T15:58:56.820636+0000","last_fullsized":"2024-03-18T15:58:56.820636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:16.894313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573231+0000","last_change":"2024-03-18T15:57:57.192612+0000","last_active":"2024-03-18T15:58:56.573231+0000","last_peered":"2024-03-18T15:58:56.573231+0000","last_clean":"2024-03-18T15:58:56.573231+0000","last_became_active":"2024-03-18T15:57:57.192284+0000","last_became_peered":"2024-03-18T15:57:57.192284+0000","last_unstale":"2024-03-18T15:58:56.573231+0000","last_undegraded":"2024-03-18T15:58:56.573231+0000","last_fullsized":"2024-03-18T15:58:56.573231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:53.597362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.580958+0000","last_change":"2024-03-18T15:57:56.165164+0000","last_active":"2024-03-18T15:58:56.580958+0000","last_peered":"2024-03-18T15:58:56.580958+0000","last_clean":"2024-03-18T15:58:56.580958+0000","last_became_active":"2024-03-18T15:57:56.164844+0000","last_became_peered":"2024-03-18T15:57:56.164844+0000","last_unstale":"2024-03-18T15:58:56.580958+0000","last_undegraded":"2024-03-18T15:58:56.580958+0000","last_fullsized":"2024-03-18T15:58:56.580958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:36.754823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038173+0000","last_change":"2024-03-18T15:57:57.180176+0000","last_active":"2024-03-18T15:58:57.038173+0000","last_peered":"2024-03-18T15:58:57.038173+0000","last_clean":"2024-03-18T15:58:57.038173+0000","last_became_active":"2024-03-18T15:57:57.179832+0000","last_became_peered":"2024-03-18T15:57:57.179832+0000","last_unstale":"2024-03-18T15:58:57.038173+0000","last_undegraded":"2024-03-18T15:58:57.038173+0000","last_fullsized":"2024-03-18T15:58:57.038173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:32.249126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573371+0000","last_change":"2024-03-18T15:57:56.175404+0000","last_active":"2024-03-18T15:58:56.573371+0000","last_peered":"2024-03-18T15:58:56.573371+0000","last_clean":"2024-03-18T15:58:56.573371+0000","last_became_active":"2024-03-18T15:57:56.174877+0000","last_became_peered":"2024-03-18T15:57:56.174877+0000","last_unstale":"2024-03-18T15:58:56.573371+0000","last_undegraded":"2024-03-18T15:58:56.573371+0000","last_fullsized":"2024-03-18T15:58:56.573371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:39.536240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568655+0000","last_change":"2024-03-18T15:57:57.178213+0000","last_active":"2024-03-18T15:58:56.568655+0000","last_peered":"2024-03-18T15:58:56.568655+0000","last_clean":"2024-03-18T15:58:56.568655+0000","last_became_active":"2024-03-18T15:57:57.177863+0000","last_became_peered":"2024-03-18T15:57:57.177863+0000","last_unstale":"2024-03-18T15:58:56.568655+0000","last_undegraded":"2024-03-18T15:58:56.568655+0000","last_fullsized":"2024-03-18T15:58:56.568655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:42.988890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:08.271040+0000","last_change":"2024-03-18T15:57:56.172506+0000","last_active":"2024-03-18T15:59:08.271040+0000","last_peered":"2024-03-18T15:59:08.271040+0000","last_clean":"2024-03-18T15:59:08.271040+0000","last_became_active":"2024-03-18T15:57:56.172153+0000","last_became_peered":"2024-03-18T15:57:56.172153+0000","last_unstale":"2024-03-18T15:59:08.271040+0000","last_undegraded":"2024-03-18T15:59:08.271040+0000","last_fullsized":"2024-03-18T15:59:08.271040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:30.563167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572556+0000","last_change":"2024-03-18T15:57:57.181506+0000","last_active":"2024-03-18T15:58:56.572556+0000","last_peered":"2024-03-18T15:58:56.572556+0000","last_clean":"2024-03-18T15:58:56.572556+0000","last_became_active":"2024-03-18T15:57:57.181139+0000","last_became_peered":"2024-03-18T15:57:57.181139+0000","last_unstale":"2024-03-18T15:58:56.572556+0000","last_undegraded":"2024-03-18T15:58:56.572556+0000","last_fullsized":"2024-03-18T15:58:56.572556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:40.239719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.771996+0000","last_change":"2024-03-18T15:58:56.771996+0000","last_active":"2024-03-18T15:58:56.771996+0000","last_peered":"2024-03-18T15:58:56.771996+0000","last_clean":"2024-03-18T15:58:56.771996+0000","last_became_active":"2024-03-18T15:58:56.769351+0000","last_became_peered":"2024-03-18T15:58:56.769351+0000","last_unstale":"2024-03-18T15:58:56.771996+0000","last_undegraded":"2024-03-18T15:58:56.771996+0000","last_fullsized":"2024-03-18T15:58:56.771996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:00.300021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820715+0000","last_change":"2024-03-18T15:57:57.186766+0000","last_active":"2024-03-18T15:58:56.820715+0000","last_peered":"2024-03-18T15:58:56.820715+0000","last_clean":"2024-03-18T15:58:56.820715+0000","last_became_active":"2024-03-18T15:57:57.186453+0000","last_became_peered":"2024-03-18T15:57:57.186453+0000","last_unstale":"2024-03-18T15:58:56.820715+0000","last_undegraded":"2024-03-18T15:58:56.820715+0000","last_fullsized":"2024-03-18T15:58:56.820715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:19.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573672+0000","last_change":"2024-03-18T15:57:56.174536+0000","last_active":"2024-03-18T15:58:56.573672+0000","last_peered":"2024-03-18T15:58:56.573672+0000","last_clean":"2024-03-18T15:58:56.573672+0000","last_became_active":"2024-03-18T15:57:56.174216+0000","last_became_peered":"2024-03-18T15:57:56.174216+0000","last_unstale":"2024-03-18T15:58:56.573672+0000","last_undegraded":"2024-03-18T15:58:56.573672+0000","last_fullsized":"2024-03-18T15:58:56.573672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:50.324174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572582+0000","last_change":"2024-03-18T15:57:57.178511+0000","last_active":"2024-03-18T15:58:56.572582+0000","last_peered":"2024-03-18T15:58:56.572582+0000","last_clean":"2024-03-18T15:58:56.572582+0000","last_became_active":"2024-03-18T15:57:57.178117+0000","last_became_peered":"2024-03-18T15:57:57.178117+0000","last_unstale":"2024-03-18T15:58:56.572582+0000","last_undegraded":"2024-03-18T15:58:56.572582+0000","last_fullsized":"2024-03-18T15:58:56.572582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:52.891086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'30","reported_seq":195,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:09.272090+0000","last_change":"2024-03-18T15:57:56.173372+0000","last_active":"2024-03-18T15:59:09.272090+0000","last_peered":"2024-03-18T15:59:09.272090+0000","last_clean":"2024-03-18T15:59:09.272090+0000","last_became_active":"2024-03-18T15:57:56.173049+0000","last_became_peered":"2024-03-18T15:57:56.173049+0000","last_unstale":"2024-03-18T15:59:09.272090+0000","last_undegraded":"2024-03-18T15:59:09.272090+0000","last_fullsized":"2024-03-18T15:59:09.272090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:01.655747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":130,"num_write":31,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574763+0000","last_change":"2024-03-18T15:57:57.183725+0000","last_active":"2024-03-18T15:58:56.574763+0000","last_peered":"2024-03-18T15:58:56.574763+0000","last_clean":"2024-03-18T15:58:56.574763+0000","last_became_active":"2024-03-18T15:57:57.183391+0000","last_became_peered":"2024-03-18T15:57:57.183391+0000","last_unstale":"2024-03-18T15:58:56.574763+0000","last_undegraded":"2024-03-18T15:58:56.574763+0000","last_fullsized":"2024-03-18T15:58:56.574763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:42.945519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572633+0000","last_change":"2024-03-18T15:57:56.179534+0000","last_active":"2024-03-18T15:58:56.572633+0000","last_peered":"2024-03-18T15:58:56.572633+0000","last_clean":"2024-03-18T15:58:56.572633+0000","last_became_active":"2024-03-18T15:57:56.179217+0000","last_became_peered":"2024-03-18T15:57:56.179217+0000","last_unstale":"2024-03-18T15:58:56.572633+0000","last_undegraded":"2024-03-18T15:58:56.572633+0000","last_fullsized":"2024-03-18T15:58:56.572633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:19.176423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581245+0000","last_change":"2024-03-18T15:57:57.188958+0000","last_active":"2024-03-18T15:58:56.581245+0000","last_peered":"2024-03-18T15:58:56.581245+0000","last_clean":"2024-03-18T15:58:56.581245+0000","last_became_active":"2024-03-18T15:57:57.188586+0000","last_became_peered":"2024-03-18T15:57:57.188586+0000","last_unstale":"2024-03-18T15:58:56.581245+0000","last_undegraded":"2024-03-18T15:58:56.581245+0000","last_fullsized":"2024-03-18T15:58:56.581245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:48.122539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572531+0000","last_change":"2024-03-18T15:57:56.180530+0000","last_active":"2024-03-18T15:58:56.572531+0000","last_peered":"2024-03-18T15:58:56.572531+0000","last_clean":"2024-03-18T15:58:56.572531+0000","last_became_active":"2024-03-18T15:57:56.180019+0000","last_became_peered":"2024-03-18T15:57:56.180019+0000","last_unstale":"2024-03-18T15:58:56.572531+0000","last_undegraded":"2024-03-18T15:58:56.572531+0000","last_fullsized":"2024-03-18T15:58:56.572531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:04.119730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581996+0000","last_change":"2024-03-18T15:58:56.581996+0000","last_active":"2024-03-18T15:58:56.581996+0000","last_peered":"2024-03-18T15:58:56.581996+0000","last_clean":"2024-03-18T15:58:56.581996+0000","last_became_active":"2024-03-18T15:58:56.581728+0000","last_became_peered":"2024-03-18T15:58:56.581728+0000","last_unstale":"2024-03-18T15:58:56.581996+0000","last_undegraded":"2024-03-18T15:58:56.581996+0000","last_fullsized":"2024-03-18T15:58:56.581996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:25.718628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:09.281941+0000","last_change":"2024-03-18T15:57:56.172587+0000","last_active":"2024-03-18T15:59:09.281941+0000","last_peered":"2024-03-18T15:59:09.281941+0000","last_clean":"2024-03-18T15:59:09.281941+0000","last_became_active":"2024-03-18T15:57:56.172246+0000","last_became_peered":"2024-03-18T15:57:56.172246+0000","last_unstale":"2024-03-18T15:59:09.281941+0000","last_undegraded":"2024-03-18T15:59:09.281941+0000","last_fullsized":"2024-03-18T15:59:09.281941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.543059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569269+0000","last_change":"2024-03-18T15:57:57.181899+0000","last_active":"2024-03-18T15:58:56.569269+0000","last_peered":"2024-03-18T15:58:56.569269+0000","last_clean":"2024-03-18T15:58:56.569269+0000","last_became_active":"2024-03-18T15:57:57.181559+0000","last_became_peered":"2024-03-18T15:57:57.181559+0000","last_unstale":"2024-03-18T15:58:56.569269+0000","last_undegraded":"2024-03-18T15:58:56.569269+0000","last_fullsized":"2024-03-18T15:58:56.569269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:17.504153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573770+0000","last_change":"2024-03-18T15:57:56.179375+0000","last_active":"2024-03-18T15:58:56.573770+0000","last_peered":"2024-03-18T15:58:56.573770+0000","last_clean":"2024-03-18T15:58:56.573770+0000","last_became_active":"2024-03-18T15:57:56.178990+0000","last_became_peered":"2024-03-18T15:57:56.178990+0000","last_unstale":"2024-03-18T15:58:56.573770+0000","last_undegraded":"2024-03-18T15:58:56.573770+0000","last_fullsized":"2024-03-18T15:58:56.573770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:36.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572251+0000","last_change":"2024-03-18T15:57:57.188135+0000","last_active":"2024-03-18T15:58:56.572251+0000","last_peered":"2024-03-18T15:58:56.572251+0000","last_clean":"2024-03-18T15:58:56.572251+0000","last_became_active":"2024-03-18T15:57:57.187811+0000","last_became_peered":"2024-03-18T15:57:57.187811+0000","last_unstale":"2024-03-18T15:58:56.572251+0000","last_undegraded":"2024-03-18T15:58:56.572251+0000","last_fullsized":"2024-03-18T15:58:56.572251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:11.235420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574600+0000","last_change":"2024-03-18T15:57:56.181774+0000","last_active":"2024-03-18T15:58:56.574600+0000","last_peered":"2024-03-18T15:58:56.574600+0000","last_clean":"2024-03-18T15:58:56.574600+0000","last_became_active":"2024-03-18T15:57:56.181360+0000","last_became_peered":"2024-03-18T15:57:56.181360+0000","last_unstale":"2024-03-18T15:58:56.574600+0000","last_undegraded":"2024-03-18T15:58:56.574600+0000","last_fullsized":"2024-03-18T15:58:56.574600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:08.367151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570042+0000","last_change":"2024-03-18T15:57:57.188960+0000","last_active":"2024-03-18T15:58:56.570042+0000","last_peered":"2024-03-18T15:58:56.570042+0000","last_clean":"2024-03-18T15:58:56.570042+0000","last_became_active":"2024-03-18T15:57:57.187528+0000","last_became_peered":"2024-03-18T15:57:57.187528+0000","last_unstale":"2024-03-18T15:58:56.570042+0000","last_undegraded":"2024-03-18T15:58:56.570042+0000","last_fullsized":"2024-03-18T15:58:56.570042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:29.259977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.821606+0000","last_change":"2024-03-18T15:58:56.821606+0000","last_active":"2024-03-18T15:58:56.821606+0000","last_peered":"2024-03-18T15:58:56.821606+0000","last_clean":"2024-03-18T15:58:56.821606+0000","last_became_active":"2024-03-18T15:58:56.821283+0000","last_became_peered":"2024-03-18T15:58:56.821283+0000","last_unstale":"2024-03-18T15:58:56.821606+0000","last_undegraded":"2024-03-18T15:58:56.821606+0000","last_fullsized":"2024-03-18T15:58:56.821606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:02.895290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820973+0000","last_change":"2024-03-18T15:57:57.189238+0000","last_active":"2024-03-18T15:58:56.820973+0000","last_peered":"2024-03-18T15:58:56.820973+0000","last_clean":"2024-03-18T15:58:56.820973+0000","last_became_active":"2024-03-18T15:57:57.188900+0000","last_became_peered":"2024-03-18T15:57:57.188900+0000","last_unstale":"2024-03-18T15:58:56.820973+0000","last_undegraded":"2024-03-18T15:58:56.820973+0000","last_fullsized":"2024-03-18T15:58:56.820973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:16.810976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569598+0000","last_change":"2024-03-18T15:57:56.178765+0000","last_active":"2024-03-18T15:58:56.569598+0000","last_peered":"2024-03-18T15:58:56.569598+0000","last_clean":"2024-03-18T15:58:56.569598+0000","last_became_active":"2024-03-18T15:57:56.178399+0000","last_became_peered":"2024-03-18T15:57:56.178399+0000","last_unstale":"2024-03-18T15:58:56.569598+0000","last_undegraded":"2024-03-18T15:58:56.569598+0000","last_fullsized":"2024-03-18T15:58:56.569598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:46:07.844453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820928+0000","last_change":"2024-03-18T15:57:57.173968+0000","last_active":"2024-03-18T15:58:56.820928+0000","last_peered":"2024-03-18T15:58:56.820928+0000","last_clean":"2024-03-18T15:58:56.820928+0000","last_became_active":"2024-03-18T15:57:57.173605+0000","last_became_peered":"2024-03-18T15:57:57.173605+0000","last_unstale":"2024-03-18T15:58:56.820928+0000","last_undegraded":"2024-03-18T15:58:56.820928+0000","last_fullsized":"2024-03-18T15:58:56.820928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:34.676393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573566+0000","last_change":"2024-03-18T15:57:57.186561+0000","last_active":"2024-03-18T15:58:56.573566+0000","last_peered":"2024-03-18T15:58:56.573566+0000","last_clean":"2024-03-18T15:58:56.573566+0000","last_became_active":"2024-03-18T15:57:57.186260+0000","last_became_peered":"2024-03-18T15:57:57.186260+0000","last_unstale":"2024-03-18T15:58:56.573566+0000","last_undegraded":"2024-03-18T15:58:56.573566+0000","last_fullsized":"2024-03-18T15:58:56.573566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:54:06.646480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:10.282488+0000","last_change":"2024-03-18T15:57:56.164175+0000","last_active":"2024-03-18T15:59:10.282488+0000","last_peered":"2024-03-18T15:59:10.282488+0000","last_clean":"2024-03-18T15:59:10.282488+0000","last_became_active":"2024-03-18T15:57:56.163724+0000","last_became_peered":"2024-03-18T15:57:56.163724+0000","last_unstale":"2024-03-18T15:59:10.282488+0000","last_undegraded":"2024-03-18T15:59:10.282488+0000","last_fullsized":"2024-03-18T15:59:10.282488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:48:53.596211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038203+0000","last_change":"2024-03-18T15:57:57.191507+0000","last_active":"2024-03-18T15:58:57.038203+0000","last_peered":"2024-03-18T15:58:57.038203+0000","last_clean":"2024-03-18T15:58:57.038203+0000","last_became_active":"2024-03-18T15:57:57.191182+0000","last_became_peered":"2024-03-18T15:57:57.191182+0000","last_unstale":"2024-03-18T15:58:57.038203+0000","last_undegraded":"2024-03-18T15:58:57.038203+0000","last_fullsized":"2024-03-18T15:58:57.038203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:39.878675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768679+0000","last_change":"2024-03-18T15:57:56.169490+0000","last_active":"2024-03-18T15:58:56.768679+0000","last_peered":"2024-03-18T15:58:56.768679+0000","last_clean":"2024-03-18T15:58:56.768679+0000","last_became_active":"2024-03-18T15:57:56.169107+0000","last_became_peered":"2024-03-18T15:57:56.169107+0000","last_unstale":"2024-03-18T15:58:56.768679+0000","last_undegraded":"2024-03-18T15:58:56.768679+0000","last_fullsized":"2024-03-18T15:58:56.768679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:59.722909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569340+0000","last_change":"2024-03-18T15:56:43.270716+0000","last_active":"2024-03-18T15:58:56.569340+0000","last_peered":"2024-03-18T15:58:56.569340+0000","last_clean":"2024-03-18T15:58:56.569340+0000","last_became_active":"2024-03-18T15:56:43.270316+0000","last_became_peered":"2024-03-18T15:56:43.270316+0000","last_unstale":"2024-03-18T15:58:56.569340+0000","last_undegraded":"2024-03-18T15:58:56.569340+0000","last_fullsized":"2024-03-18T15:58:56.569340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_clean_scrub_stamp":"2024-03-18T15:53:17.916048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:00.779948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.580856+0000","last_change":"2024-03-18T15:57:57.172265+0000","last_active":"2024-03-18T15:58:56.580856+0000","last_peered":"2024-03-18T15:58:56.580856+0000","last_clean":"2024-03-18T15:58:56.580856+0000","last_became_active":"2024-03-18T15:57:57.171962+0000","last_became_peered":"2024-03-18T15:57:57.171962+0000","last_unstale":"2024-03-18T15:58:56.580856+0000","last_undegraded":"2024-03-18T15:58:56.580856+0000","last_fullsized":"2024-03-18T15:58:56.580856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:41:44.402012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.577920+0000","last_change":"2024-03-18T15:58:56.577920+0000","last_active":"2024-03-18T15:58:56.577920+0000","last_peered":"2024-03-18T15:58:56.577920+0000","last_clean":"2024-03-18T15:58:56.577920+0000","last_became_active":"2024-03-18T15:58:56.574742+0000","last_became_peered":"2024-03-18T15:58:56.574742+0000","last_unstale":"2024-03-18T15:58:56.577920+0000","last_undegraded":"2024-03-18T15:58:56.577920+0000","last_fullsized":"2024-03-18T15:58:56.577920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:00.046050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574865+0000","last_change":"2024-03-18T15:57:56.178710+0000","last_active":"2024-03-18T15:58:56.574865+0000","last_peered":"2024-03-18T15:58:56.574865+0000","last_clean":"2024-03-18T15:58:56.574865+0000","last_became_active":"2024-03-18T15:57:56.178388+0000","last_became_peered":"2024-03-18T15:57:56.178388+0000","last_unstale":"2024-03-18T15:58:56.574865+0000","last_undegraded":"2024-03-18T15:58:56.574865+0000","last_fullsized":"2024-03-18T15:58:56.574865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:23.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581151+0000","last_change":"2024-03-18T15:57:57.168951+0000","last_active":"2024-03-18T15:58:56.581151+0000","last_peered":"2024-03-18T15:58:56.581151+0000","last_clean":"2024-03-18T15:58:56.581151+0000","last_became_active":"2024-03-18T15:57:57.168567+0000","last_became_peered":"2024-03-18T15:57:57.168567+0000","last_unstale":"2024-03-18T15:58:56.581151+0000","last_undegraded":"2024-03-18T15:58:56.581151+0000","last_fullsized":"2024-03-18T15:58:56.581151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:41.171254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569059+0000","last_change":"2024-03-18T15:57:56.171665+0000","last_active":"2024-03-18T15:58:56.569059+0000","last_peered":"2024-03-18T15:58:56.569059+0000","last_clean":"2024-03-18T15:58:56.569059+0000","last_became_active":"2024-03-18T15:57:56.171282+0000","last_became_peered":"2024-03-18T15:57:56.171282+0000","last_unstale":"2024-03-18T15:58:56.569059+0000","last_undegraded":"2024-03-18T15:58:56.569059+0000","last_fullsized":"2024-03-18T15:58:56.569059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:03:27.523561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038327+0000","last_change":"2024-03-18T15:57:57.185647+0000","last_active":"2024-03-18T15:58:57.038327+0000","last_peered":"2024-03-18T15:58:57.038327+0000","last_clean":"2024-03-18T15:58:57.038327+0000","last_became_active":"2024-03-18T15:57:57.185352+0000","last_became_peered":"2024-03-18T15:57:57.185352+0000","last_unstale":"2024-03-18T15:58:57.038327+0000","last_undegraded":"2024-03-18T15:58:57.038327+0000","last_fullsized":"2024-03-18T15:58:57.038327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:38.247776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768809+0000","last_change":"2024-03-18T15:57:56.170878+0000","last_active":"2024-03-18T15:58:56.768809+0000","last_peered":"2024-03-18T15:58:56.768809+0000","last_clean":"2024-03-18T15:58:56.768809+0000","last_became_active":"2024-03-18T15:57:56.170558+0000","last_became_peered":"2024-03-18T15:57:56.170558+0000","last_unstale":"2024-03-18T15:58:56.768809+0000","last_undegraded":"2024-03-18T15:58:56.768809+0000","last_fullsized":"2024-03-18T15:58:56.768809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:26:39.963384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820621+0000","last_change":"2024-03-18T15:57:57.181012+0000","last_active":"2024-03-18T15:58:56.820621+0000","last_peered":"2024-03-18T15:58:56.820621+0000","last_clean":"2024-03-18T15:58:56.820621+0000","last_became_active":"2024-03-18T15:57:57.180613+0000","last_became_peered":"2024-03-18T15:57:57.180613+0000","last_unstale":"2024-03-18T15:58:56.820621+0000","last_undegraded":"2024-03-18T15:58:56.820621+0000","last_fullsized":"2024-03-18T15:58:56.820621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:22.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572378+0000","last_change":"2024-03-18T15:57:56.171929+0000","last_active":"2024-03-18T15:58:56.572378+0000","last_peered":"2024-03-18T15:58:56.572378+0000","last_clean":"2024-03-18T15:58:56.572378+0000","last_became_active":"2024-03-18T15:57:56.171570+0000","last_became_peered":"2024-03-18T15:57:56.171570+0000","last_unstale":"2024-03-18T15:58:56.572378+0000","last_undegraded":"2024-03-18T15:58:56.572378+0000","last_fullsized":"2024-03-18T15:58:56.572378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:11.579956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573376+0000","last_change":"2024-03-18T15:57:57.178118+0000","last_active":"2024-03-18T15:58:56.573376+0000","last_peered":"2024-03-18T15:58:56.573376+0000","last_clean":"2024-03-18T15:58:56.573376+0000","last_became_active":"2024-03-18T15:57:57.177764+0000","last_became_peered":"2024-03-18T15:57:57.177764+0000","last_unstale":"2024-03-18T15:58:56.573376+0000","last_undegraded":"2024-03-18T15:58:56.573376+0000","last_fullsized":"2024-03-18T15:58:56.573376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:07.519544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038062+0000","last_change":"2024-03-18T15:57:56.168485+0000","last_active":"2024-03-18T15:58:57.038062+0000","last_peered":"2024-03-18T15:58:57.038062+0000","last_clean":"2024-03-18T15:58:57.038062+0000","last_became_active":"2024-03-18T15:57:56.168124+0000","last_became_peered":"2024-03-18T15:57:56.168124+0000","last_unstale":"2024-03-18T15:58:57.038062+0000","last_undegraded":"2024-03-18T15:58:57.038062+0000","last_fullsized":"2024-03-18T15:58:57.038062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:09.465975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569757+0000","last_change":"2024-03-18T15:57:57.184781+0000","last_active":"2024-03-18T15:58:56.569757+0000","last_peered":"2024-03-18T15:58:56.569757+0000","last_clean":"2024-03-18T15:58:56.569757+0000","last_became_active":"2024-03-18T15:57:57.184431+0000","last_became_peered":"2024-03-18T15:57:57.184431+0000","last_unstale":"2024-03-18T15:58:56.569757+0000","last_undegraded":"2024-03-18T15:58:56.569757+0000","last_fullsized":"2024-03-18T15:58:56.569757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:32.945288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572620+0000","last_change":"2024-03-18T15:57:56.180295+0000","last_active":"2024-03-18T15:58:56.572620+0000","last_peered":"2024-03-18T15:58:56.572620+0000","last_clean":"2024-03-18T15:58:56.572620+0000","last_became_active":"2024-03-18T15:57:56.179956+0000","last_became_peered":"2024-03-18T15:57:56.179956+0000","last_unstale":"2024-03-18T15:58:56.572620+0000","last_undegraded":"2024-03-18T15:58:56.572620+0000","last_fullsized":"2024-03-18T15:58:56.572620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:49.664982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038445+0000","last_change":"2024-03-18T15:57:57.187421+0000","last_active":"2024-03-18T15:58:57.038445+0000","last_peered":"2024-03-18T15:58:57.038445+0000","last_clean":"2024-03-18T15:58:57.038445+0000","last_became_active":"2024-03-18T15:57:57.185978+0000","last_became_peered":"2024-03-18T15:57:57.185978+0000","last_unstale":"2024-03-18T15:58:57.038445+0000","last_undegraded":"2024-03-18T15:58:57.038445+0000","last_fullsized":"2024-03-18T15:58:57.038445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:07:59.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574336+0000","last_change":"2024-03-18T15:57:56.178118+0000","last_active":"2024-03-18T15:58:56.574336+0000","last_peered":"2024-03-18T15:58:56.574336+0000","last_clean":"2024-03-18T15:58:56.574336+0000","last_became_active":"2024-03-18T15:57:56.177758+0000","last_became_peered":"2024-03-18T15:57:56.177758+0000","last_unstale":"2024-03-18T15:58:56.574336+0000","last_undegraded":"2024-03-18T15:58:56.574336+0000","last_fullsized":"2024-03-18T15:58:56.574336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:34.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570140+0000","last_change":"2024-03-18T15:57:57.182666+0000","last_active":"2024-03-18T15:58:56.570140+0000","last_peered":"2024-03-18T15:58:56.570140+0000","last_clean":"2024-03-18T15:58:56.570140+0000","last_became_active":"2024-03-18T15:57:57.182328+0000","last_became_peered":"2024-03-18T15:57:57.182328+0000","last_unstale":"2024-03-18T15:58:56.570140+0000","last_undegraded":"2024-03-18T15:58:56.570140+0000","last_fullsized":"2024-03-18T15:58:56.570140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:55.132441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573040+0000","last_change":"2024-03-18T15:57:56.177101+0000","last_active":"2024-03-18T15:58:56.573040+0000","last_peered":"2024-03-18T15:58:56.573040+0000","last_clean":"2024-03-18T15:58:56.573040+0000","last_became_active":"2024-03-18T15:57:56.176734+0000","last_became_peered":"2024-03-18T15:57:56.176734+0000","last_unstale":"2024-03-18T15:58:56.573040+0000","last_undegraded":"2024-03-18T15:58:56.573040+0000","last_fullsized":"2024-03-18T15:58:56.573040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:55.406210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569787+0000","last_change":"2024-03-18T15:57:57.183892+0000","last_active":"2024-03-18T15:58:56.569787+0000","last_peered":"2024-03-18T15:58:56.569787+0000","last_clean":"2024-03-18T15:58:56.569787+0000","last_became_active":"2024-03-18T15:57:57.183121+0000","last_became_peered":"2024-03-18T15:57:57.183121+0000","last_unstale":"2024-03-18T15:58:56.569787+0000","last_undegraded":"2024-03-18T15:58:56.569787+0000","last_fullsized":"2024-03-18T15:58:56.569787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:11:13.504169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573457+0000","last_change":"2024-03-18T15:57:56.172479+0000","last_active":"2024-03-18T15:58:56.573457+0000","last_peered":"2024-03-18T15:58:56.573457+0000","last_clean":"2024-03-18T15:58:56.573457+0000","last_became_active":"2024-03-18T15:57:56.172120+0000","last_became_peered":"2024-03-18T15:57:56.172120+0000","last_unstale":"2024-03-18T15:58:56.573457+0000","last_undegraded":"2024-03-18T15:58:56.573457+0000","last_fullsized":"2024-03-18T15:58:56.573457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:15.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569645+0000","last_change":"2024-03-18T15:57:57.183674+0000","last_active":"2024-03-18T15:58:56.569645+0000","last_peered":"2024-03-18T15:58:56.569645+0000","last_clean":"2024-03-18T15:58:56.569645+0000","last_became_active":"2024-03-18T15:57:57.183270+0000","last_became_peered":"2024-03-18T15:57:57.183270+0000","last_unstale":"2024-03-18T15:58:56.569645+0000","last_undegraded":"2024-03-18T15:58:56.569645+0000","last_fullsized":"2024-03-18T15:58:56.569645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:43.258995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581093+0000","last_change":"2024-03-18T15:57:56.174497+0000","last_active":"2024-03-18T15:58:56.581093+0000","last_peered":"2024-03-18T15:58:56.581093+0000","last_clean":"2024-03-18T15:58:56.581093+0000","last_became_active":"2024-03-18T15:57:56.173862+0000","last_became_peered":"2024-03-18T15:57:56.173862+0000","last_unstale":"2024-03-18T15:58:56.581093+0000","last_undegraded":"2024-03-18T15:58:56.581093+0000","last_fullsized":"2024-03-18T15:58:56.581093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:00.362276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569561+0000","last_change":"2024-03-18T15:57:57.179510+0000","last_active":"2024-03-18T15:58:56.569561+0000","last_peered":"2024-03-18T15:58:56.569561+0000","last_clean":"2024-03-18T15:58:56.569561+0000","last_became_active":"2024-03-18T15:57:57.179178+0000","last_became_peered":"2024-03-18T15:57:57.179178+0000","last_unstale":"2024-03-18T15:58:56.569561+0000","last_undegraded":"2024-03-18T15:58:56.569561+0000","last_fullsized":"2024-03-18T15:58:56.569561+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:46.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569814+0000","last_change":"2024-03-18T15:57:57.186842+0000","last_active":"2024-03-18T15:58:56.569814+0000","last_peered":"2024-03-18T15:58:56.569814+0000","last_clean":"2024-03-18T15:58:56.569814+0000","last_became_active":"2024-03-18T15:57:57.186501+0000","last_became_peered":"2024-03-18T15:57:57.186501+0000","last_unstale":"2024-03-18T15:58:56.569814+0000","last_undegraded":"2024-03-18T15:58:56.569814+0000","last_fullsized":"2024-03-18T15:58:56.569814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:11.988494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572387+0000","last_change":"2024-03-18T15:57:57.177630+0000","last_active":"2024-03-18T15:58:56.572387+0000","last_peered":"2024-03-18T15:58:56.572387+0000","last_clean":"2024-03-18T15:58:56.572387+0000","last_became_active":"2024-03-18T15:57:57.177227+0000","last_became_peered":"2024-03-18T15:57:57.177227+0000","last_unstale":"2024-03-18T15:58:56.572387+0000","last_undegraded":"2024-03-18T15:58:56.572387+0000","last_fullsized":"2024-03-18T15:58:56.572387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:27.004129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572543+0000","last_change":"2024-03-18T15:57:57.189048+0000","last_active":"2024-03-18T15:58:56.572543+0000","last_peered":"2024-03-18T15:58:56.572543+0000","last_clean":"2024-03-18T15:58:56.572543+0000","last_became_active":"2024-03-18T15:57:57.188704+0000","last_became_peered":"2024-03-18T15:57:57.188704+0000","last_unstale":"2024-03-18T15:58:56.572543+0000","last_undegraded":"2024-03-18T15:58:56.572543+0000","last_fullsized":"2024-03-18T15:58:56.572543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:10:53.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820482+0000","last_change":"2024-03-18T15:57:57.178611+0000","last_active":"2024-03-18T15:58:56.820482+0000","last_peered":"2024-03-18T15:58:56.820482+0000","last_clean":"2024-03-18T15:58:56.820482+0000","last_became_active":"2024-03-18T15:57:57.178305+0000","last_became_peered":"2024-03-18T15:57:57.178305+0000","last_unstale":"2024-03-18T15:58:56.820482+0000","last_undegraded":"2024-03-18T15:58:56.820482+0000","last_fullsized":"2024-03-18T15:58:56.820482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:51.925278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581087+0000","last_change":"2024-03-18T15:57:57.177928+0000","last_active":"2024-03-18T15:58:56.581087+0000","last_peered":"2024-03-18T15:58:56.581087+0000","last_clean":"2024-03-18T15:58:56.581087+0000","last_became_active":"2024-03-18T15:57:57.177364+0000","last_became_peered":"2024-03-18T15:57:57.177364+0000","last_unstale":"2024-03-18T15:58:56.581087+0000","last_undegraded":"2024-03-18T15:58:56.581087+0000","last_fullsized":"2024-03-18T15:58:56.581087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:59.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570620+0000","last_change":"2024-03-18T15:57:57.188080+0000","last_active":"2024-03-18T15:58:56.570620+0000","last_peered":"2024-03-18T15:58:56.570620+0000","last_clean":"2024-03-18T15:58:56.570620+0000","last_became_active":"2024-03-18T15:57:57.187767+0000","last_became_peered":"2024-03-18T15:57:57.187767+0000","last_unstale":"2024-03-18T15:58:56.570620+0000","last_undegraded":"2024-03-18T15:58:56.570620+0000","last_fullsized":"2024-03-18T15:58:56.570620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:26.290262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572953+0000","last_change":"2024-03-18T15:57:57.178581+0000","last_active":"2024-03-18T15:58:56.572953+0000","last_peered":"2024-03-18T15:58:56.572953+0000","last_clean":"2024-03-18T15:58:56.572953+0000","last_became_active":"2024-03-18T15:57:57.178168+0000","last_became_peered":"2024-03-18T15:57:57.178168+0000","last_unstale":"2024-03-18T15:58:56.572953+0000","last_undegraded":"2024-03-18T15:58:56.572953+0000","last_fullsized":"2024-03-18T15:58:56.572953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:07:07.601959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038309+0000","last_change":"2024-03-18T15:57:57.188709+0000","last_active":"2024-03-18T15:58:57.038309+0000","last_peered":"2024-03-18T15:58:57.038309+0000","last_clean":"2024-03-18T15:58:57.038309+0000","last_became_active":"2024-03-18T15:57:57.187884+0000","last_became_peered":"2024-03-18T15:57:57.187884+0000","last_unstale":"2024-03-18T15:58:57.038309+0000","last_undegraded":"2024-03-18T15:58:57.038309+0000","last_fullsized":"2024-03-18T15:58:57.038309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:19.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573475+0000","last_change":"2024-03-18T15:57:57.182489+0000","last_active":"2024-03-18T15:58:56.573475+0000","last_peered":"2024-03-18T15:58:56.573475+0000","last_clean":"2024-03-18T15:58:56.573475+0000","last_became_active":"2024-03-18T15:57:57.176462+0000","last_became_peered":"2024-03-18T15:57:57.176462+0000","last_unstale":"2024-03-18T15:58:56.573475+0000","last_undegraded":"2024-03-18T15:58:56.573475+0000","last_fullsized":"2024-03-18T15:58:56.573475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:46.138422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038009+0000","last_change":"2024-03-18T15:57:57.185848+0000","last_active":"2024-03-18T15:58:57.038009+0000","last_peered":"2024-03-18T15:58:57.038009+0000","last_clean":"2024-03-18T15:58:57.038009+0000","last_became_active":"2024-03-18T15:57:57.185507+0000","last_became_peered":"2024-03-18T15:57:57.185507+0000","last_unstale":"2024-03-18T15:58:57.038009+0000","last_undegraded":"2024-03-18T15:58:57.038009+0000","last_fullsized":"2024-03-18T15:58:57.038009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:09.319325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572353+0000","last_change":"2024-03-18T15:57:57.189222+0000","last_active":"2024-03-18T15:58:56.572353+0000","last_peered":"2024-03-18T15:58:56.572353+0000","last_clean":"2024-03-18T15:58:56.572353+0000","last_became_active":"2024-03-18T15:57:57.188847+0000","last_became_peered":"2024-03-18T15:57:57.188847+0000","last_unstale":"2024-03-18T15:58:56.572353+0000","last_undegraded":"2024-03-18T15:58:56.572353+0000","last_fullsized":"2024-03-18T15:58:56.572353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:15.935552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581304+0000","last_change":"2024-03-18T15:57:57.183269+0000","last_active":"2024-03-18T15:58:56.581304+0000","last_peered":"2024-03-18T15:58:56.581304+0000","last_clean":"2024-03-18T15:58:56.581304+0000","last_became_active":"2024-03-18T15:57:57.182866+0000","last_became_peered":"2024-03-18T15:57:57.182866+0000","last_unstale":"2024-03-18T15:58:56.581304+0000","last_undegraded":"2024-03-18T15:58:56.581304+0000","last_fullsized":"2024-03-18T15:58:56.581304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:15:08.500842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.577407+0000","last_change":"2024-03-18T15:58:56.577407+0000","last_active":"2024-03-18T15:58:56.577407+0000","last_peered":"2024-03-18T15:58:56.577407+0000","last_clean":"2024-03-18T15:58:56.577407+0000","last_became_active":"2024-03-18T15:58:56.577018+0000","last_became_peered":"2024-03-18T15:58:56.577018+0000","last_unstale":"2024-03-18T15:58:56.577407+0000","last_undegraded":"2024-03-18T15:58:56.577407+0000","last_fullsized":"2024-03-18T15:58:56.577407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:20.731241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285179,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":444,"num_read_kb":288,"num_write":156,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":460588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":1112092,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":460824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":460744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":1052283,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":460213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":520029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":461484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":521379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":460524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":460650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1051967,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:59:16.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T15:59:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:16.830 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:16 smithi150 ceph-mon[31946]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:16.830 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:16 smithi150 ceph-mon[31946]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:16 smithi191 ceph-mon[36784]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:16 smithi191 ceph-mon[36784]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:17.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:16 smithi156 ceph-mon[35919]: from='client.24817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:17.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:16 smithi156 ceph-mon[35919]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:18.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 -- 172.21.15.150:0/1355679454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 msgr2=0x7fbd04101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 --2- 172.21.15.150:0/1355679454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd04101630 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009990 tx=0x7fbcf4009ca0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 -- 172.21.15.150:0/1355679454 shutdown_connections 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 --2- 172.21.15.150:0/1355679454 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04101b70 0x7fbd0410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 --2- 172.21.15.150:0/1355679454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04107210 0x7fbd041075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 --2- 172.21.15.150:0/1355679454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd04101630 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 -- 172.21.15.150:0/1355679454 >> 172.21.15.150:0/1355679454 conn(0x7fbd04078100 msgr2=0x7fbd04078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 -- 172.21.15.150:0/1355679454 shutdown_connections 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.072+0000 7fbd0a547700 1 -- 172.21.15.150:0/1355679454 wait complete. 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 Processor -- start 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 -- start start 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 0x7fbd0419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 0x7fbd04198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:18.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbd04111470 con 0x7fbd041011d0 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041112f0 con 0x7fbd04101b70 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd0a547700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041115f0 con 0x7fbd04107210 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd03fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39340/0 (socket says 172.21.15.150:39340) 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd03fff700 1 -- 172.21.15.150:0/261232978 learned_addr learned my addr 172.21.15.150:0/261232978 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd08ae4700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 0x7fbd04198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd037fe700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 0x7fbd0419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd03fff700 1 -- 172.21.15.150:0/261232978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 msgr2=0x7fbd04198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.073+0000 7fbd03fff700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 0x7fbd04198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd03fff700 1 -- 172.21.15.150:0/261232978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 msgr2=0x7fbd0419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd03fff700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 0x7fbd0419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd03fff700 1 -- 172.21.15.150:0/261232978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd04198fc0 con 0x7fbd041011d0 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd08ae4700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 0x7fbd04198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd037fe700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 0x7fbd0419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:18.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd03fff700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800e9a0 tx=0x7fbcf800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:18.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800ca10 con 0x7fbd041011d0 2024-03-18T15:59:18.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd041992b0 con 0x7fbd041011d0 2024-03-18T15:59:18.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf800cbb0 con 0x7fbd041011d0 2024-03-18T15:59:18.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8018680 con 0x7fbd041011d0 2024-03-18T15:59:18.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.074+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd041a5570 con 0x7fbd041011d0 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.075+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd04068f90 con 0x7fbd041011d0 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.076+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fbcf8010430 con 0x7fbd041011d0 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.076+0000 7fbd017fa700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 0x7fbcec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.077+0000 7fbd037fe700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 0x7fbcec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.077+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbcf80ced80 con 0x7fbd041011d0 2024-03-18T15:59:18.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.077+0000 7fbd037fe700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 0x7fbcec0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400b2e0 tx=0x7fbcf4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:18.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.081+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbcf809bcf0 con 0x7fbd041011d0 2024-03-18T15:59:18.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.271+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbd04104fc0 con 0x7fbcec0a2920 2024-03-18T15:59:18.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.279+0000 7fbd017fa700 1 -- 172.21.15.150:0/261232978 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258877 (secure 0 0 0) 0x7fbcf4067350 con 0x7fbcec0a2920 2024-03-18T15:59:18.283 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 msgr2=0x7fbcec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 0x7fbcec0a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400b2e0 tx=0x7fbcf4005960 comp rx=0 tx=0).stop 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 msgr2=0x7fbd0419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800e9a0 tx=0x7fbcf800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 shutdown_connections 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd04107210 0x7fbd04198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd041011d0 0x7fbd0419e6a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbcec0a2920 0x7fbcec0a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 --2- 172.21.15.150:0/261232978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd04101b70 0x7fbd0419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:18.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.281+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 >> 172.21.15.150:0/261232978 conn(0x7fbd04078100 msgr2=0x7fbd04100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:18.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.282+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 shutdown_connections 2024-03-18T15:59:18.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:18.282+0000 7fbd0a547700 1 -- 172.21.15.150:0/261232978 wait complete. 2024-03-18T15:59:18.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-18T15:59:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:18 smithi150 ceph-mon[31946]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:18.962 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-03-18T15:59:17.463667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654235,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":610,"num_read_kb":422,"num_write":339,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377576,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564248,"statfs":{"total":1151940427776,"available":1151553789952,"internally_reserved":0,"allocated":11993088,"data_stored":7482777,"data_compressed":108786,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570404+0000","last_change":"2024-03-18T15:57:57.181734+0000","last_active":"2024-03-18T15:58:56.570404+0000","last_peered":"2024-03-18T15:58:56.570404+0000","last_clean":"2024-03-18T15:58:56.570404+0000","last_became_active":"2024-03-18T15:57:57.181321+0000","last_became_peered":"2024-03-18T15:57:57.181321+0000","last_unstale":"2024-03-18T15:58:56.570404+0000","last_undegraded":"2024-03-18T15:58:56.570404+0000","last_fullsized":"2024-03-18T15:58:56.570404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:14.921877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820601+0000","last_change":"2024-03-18T15:57:57.175040+0000","last_active":"2024-03-18T15:58:56.820601+0000","last_peered":"2024-03-18T15:58:56.820601+0000","last_clean":"2024-03-18T15:58:56.820601+0000","last_became_active":"2024-03-18T15:57:57.174721+0000","last_became_peered":"2024-03-18T15:57:57.174721+0000","last_unstale":"2024-03-18T15:58:56.820601+0000","last_undegraded":"2024-03-18T15:58:56.820601+0000","last_fullsized":"2024-03-18T15:58:56.820601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:30.844693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820544+0000","last_change":"2024-03-18T15:57:57.183409+0000","last_active":"2024-03-18T15:58:56.820544+0000","last_peered":"2024-03-18T15:58:56.820544+0000","last_clean":"2024-03-18T15:58:56.820544+0000","last_became_active":"2024-03-18T15:57:57.183089+0000","last_became_peered":"2024-03-18T15:57:57.183089+0000","last_unstale":"2024-03-18T15:58:56.820544+0000","last_undegraded":"2024-03-18T15:58:56.820544+0000","last_fullsized":"2024-03-18T15:58:56.820544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:25.702402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768596+0000","last_change":"2024-03-18T15:57:57.183104+0000","last_active":"2024-03-18T15:58:56.768596+0000","last_peered":"2024-03-18T15:58:56.768596+0000","last_clean":"2024-03-18T15:58:56.768596+0000","last_became_active":"2024-03-18T15:57:57.182750+0000","last_became_peered":"2024-03-18T15:57:57.182750+0000","last_unstale":"2024-03-18T15:58:56.768596+0000","last_undegraded":"2024-03-18T15:58:56.768596+0000","last_fullsized":"2024-03-18T15:58:56.768596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:25.369309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574242+0000","last_change":"2024-03-18T15:57:57.195272+0000","last_active":"2024-03-18T15:58:56.574242+0000","last_peered":"2024-03-18T15:58:56.574242+0000","last_clean":"2024-03-18T15:58:56.574242+0000","last_became_active":"2024-03-18T15:57:57.192605+0000","last_became_peered":"2024-03-18T15:57:57.192605+0000","last_unstale":"2024-03-18T15:58:56.574242+0000","last_undegraded":"2024-03-18T15:58:56.574242+0000","last_fullsized":"2024-03-18T15:58:56.574242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:05.719973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820641+0000","last_change":"2024-03-18T15:57:57.177918+0000","last_active":"2024-03-18T15:58:56.820641+0000","last_peered":"2024-03-18T15:58:56.820641+0000","last_clean":"2024-03-18T15:58:56.820641+0000","last_became_active":"2024-03-18T15:57:57.177551+0000","last_became_peered":"2024-03-18T15:57:57.177551+0000","last_unstale":"2024-03-18T15:58:56.820641+0000","last_undegraded":"2024-03-18T15:58:56.820641+0000","last_fullsized":"2024-03-18T15:58:56.820641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:15.033755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569391+0000","last_change":"2024-03-18T15:57:57.179306+0000","last_active":"2024-03-18T15:58:56.569391+0000","last_peered":"2024-03-18T15:58:56.569391+0000","last_clean":"2024-03-18T15:58:56.569391+0000","last_became_active":"2024-03-18T15:57:57.178955+0000","last_became_peered":"2024-03-18T15:57:57.178955+0000","last_unstale":"2024-03-18T15:58:56.569391+0000","last_undegraded":"2024-03-18T15:58:56.569391+0000","last_fullsized":"2024-03-18T15:58:56.569391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:47.343910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.822345+0000","last_change":"2024-03-18T15:58:56.821934+0000","last_active":"2024-03-18T15:58:56.822345+0000","last_peered":"2024-03-18T15:58:56.822345+0000","last_clean":"2024-03-18T15:58:56.822345+0000","last_became_active":"2024-03-18T15:58:56.821543+0000","last_became_peered":"2024-03-18T15:58:56.821543+0000","last_unstale":"2024-03-18T15:58:56.822345+0000","last_undegraded":"2024-03-18T15:58:56.822345+0000","last_fullsized":"2024-03-18T15:58:56.822345+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:06.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572683+0000","last_change":"2024-03-18T15:57:57.190203+0000","last_active":"2024-03-18T15:58:56.572683+0000","last_peered":"2024-03-18T15:58:56.572683+0000","last_clean":"2024-03-18T15:58:56.572683+0000","last_became_active":"2024-03-18T15:57:57.189850+0000","last_became_peered":"2024-03-18T15:57:57.189850+0000","last_unstale":"2024-03-18T15:58:56.572683+0000","last_undegraded":"2024-03-18T15:58:56.572683+0000","last_fullsized":"2024-03-18T15:58:56.572683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:46.274479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.821049+0000","last_change":"2024-03-18T15:57:57.171053+0000","last_active":"2024-03-18T15:58:56.821049+0000","last_peered":"2024-03-18T15:58:56.821049+0000","last_clean":"2024-03-18T15:58:56.821049+0000","last_became_active":"2024-03-18T15:57:57.170727+0000","last_became_peered":"2024-03-18T15:57:57.170727+0000","last_unstale":"2024-03-18T15:58:56.821049+0000","last_undegraded":"2024-03-18T15:58:56.821049+0000","last_fullsized":"2024-03-18T15:58:56.821049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.435390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569691+0000","last_change":"2024-03-18T15:57:57.184219+0000","last_active":"2024-03-18T15:58:56.569691+0000","last_peered":"2024-03-18T15:58:56.569691+0000","last_clean":"2024-03-18T15:58:56.569691+0000","last_became_active":"2024-03-18T15:57:57.183575+0000","last_became_peered":"2024-03-18T15:57:57.183575+0000","last_unstale":"2024-03-18T15:58:56.569691+0000","last_undegraded":"2024-03-18T15:58:56.569691+0000","last_fullsized":"2024-03-18T15:58:56.569691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:59.171634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768560+0000","last_change":"2024-03-18T15:57:57.185462+0000","last_active":"2024-03-18T15:58:56.768560+0000","last_peered":"2024-03-18T15:58:56.768560+0000","last_clean":"2024-03-18T15:58:56.768560+0000","last_became_active":"2024-03-18T15:57:57.185136+0000","last_became_peered":"2024-03-18T15:57:57.185136+0000","last_unstale":"2024-03-18T15:58:56.768560+0000","last_undegraded":"2024-03-18T15:58:56.768560+0000","last_fullsized":"2024-03-18T15:58:56.768560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:36.579297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581265+0000","last_change":"2024-03-18T15:57:57.176638+0000","last_active":"2024-03-18T15:58:56.581265+0000","last_peered":"2024-03-18T15:58:56.581265+0000","last_clean":"2024-03-18T15:58:56.581265+0000","last_became_active":"2024-03-18T15:57:57.176327+0000","last_became_peered":"2024-03-18T15:57:57.176327+0000","last_unstale":"2024-03-18T15:58:56.581265+0000","last_undegraded":"2024-03-18T15:58:56.581265+0000","last_fullsized":"2024-03-18T15:58:56.581265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:18.115306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820353+0000","last_change":"2024-03-18T15:57:57.172258+0000","last_active":"2024-03-18T15:58:56.820353+0000","last_peered":"2024-03-18T15:58:56.820353+0000","last_clean":"2024-03-18T15:58:56.820353+0000","last_became_active":"2024-03-18T15:57:57.171916+0000","last_became_peered":"2024-03-18T15:57:57.171916+0000","last_unstale":"2024-03-18T15:58:56.820353+0000","last_undegraded":"2024-03-18T15:58:56.820353+0000","last_fullsized":"2024-03-18T15:58:56.820353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:24.467000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820298+0000","last_change":"2024-03-18T15:57:57.173523+0000","last_active":"2024-03-18T15:58:56.820298+0000","last_peered":"2024-03-18T15:58:56.820298+0000","last_clean":"2024-03-18T15:58:56.820298+0000","last_became_active":"2024-03-18T15:57:57.173203+0000","last_became_peered":"2024-03-18T15:57:57.173203+0000","last_unstale":"2024-03-18T15:58:56.820298+0000","last_undegraded":"2024-03-18T15:58:56.820298+0000","last_fullsized":"2024-03-18T15:58:56.820298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:31.383153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572567+0000","last_change":"2024-03-18T15:57:57.185214+0000","last_active":"2024-03-18T15:58:56.572567+0000","last_peered":"2024-03-18T15:58:56.572567+0000","last_clean":"2024-03-18T15:58:56.572567+0000","last_became_active":"2024-03-18T15:57:57.183794+0000","last_became_peered":"2024-03-18T15:57:57.183794+0000","last_unstale":"2024-03-18T15:58:56.572567+0000","last_undegraded":"2024-03-18T15:58:56.572567+0000","last_fullsized":"2024-03-18T15:58:56.572567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:45.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570494+0000","last_change":"2024-03-18T15:57:57.183349+0000","last_active":"2024-03-18T15:58:56.570494+0000","last_peered":"2024-03-18T15:58:56.570494+0000","last_clean":"2024-03-18T15:58:56.570494+0000","last_became_active":"2024-03-18T15:57:57.182127+0000","last_became_peered":"2024-03-18T15:57:57.182127+0000","last_unstale":"2024-03-18T15:58:56.570494+0000","last_undegraded":"2024-03-18T15:58:56.570494+0000","last_fullsized":"2024-03-18T15:58:56.570494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:17.229945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568950+0000","last_change":"2024-03-18T15:57:57.182112+0000","last_active":"2024-03-18T15:58:56.568950+0000","last_peered":"2024-03-18T15:58:56.568950+0000","last_clean":"2024-03-18T15:58:56.568950+0000","last_became_active":"2024-03-18T15:57:57.181695+0000","last_became_peered":"2024-03-18T15:57:57.181695+0000","last_unstale":"2024-03-18T15:58:56.568950+0000","last_undegraded":"2024-03-18T15:58:56.568950+0000","last_fullsized":"2024-03-18T15:58:56.568950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:20.402597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572356+0000","last_change":"2024-03-18T15:57:57.182187+0000","last_active":"2024-03-18T15:58:56.572356+0000","last_peered":"2024-03-18T15:58:56.572356+0000","last_clean":"2024-03-18T15:58:56.572356+0000","last_became_active":"2024-03-18T15:57:57.181837+0000","last_became_peered":"2024-03-18T15:57:57.181837+0000","last_unstale":"2024-03-18T15:58:56.572356+0000","last_undegraded":"2024-03-18T15:58:56.572356+0000","last_fullsized":"2024-03-18T15:58:56.572356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:37:32.225077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568952+0000","last_change":"2024-03-18T15:57:56.171691+0000","last_active":"2024-03-18T15:58:56.568952+0000","last_peered":"2024-03-18T15:58:56.568952+0000","last_clean":"2024-03-18T15:58:56.568952+0000","last_became_active":"2024-03-18T15:57:56.171241+0000","last_became_peered":"2024-03-18T15:57:56.171241+0000","last_unstale":"2024-03-18T15:58:56.568952+0000","last_undegraded":"2024-03-18T15:58:56.568952+0000","last_fullsized":"2024-03-18T15:58:56.568952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:50.411828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038413+0000","last_change":"2024-03-18T15:57:57.191757+0000","last_active":"2024-03-18T15:58:57.038413+0000","last_peered":"2024-03-18T15:58:57.038413+0000","last_clean":"2024-03-18T15:58:57.038413+0000","last_became_active":"2024-03-18T15:57:57.191474+0000","last_became_peered":"2024-03-18T15:57:57.191474+0000","last_unstale":"2024-03-18T15:58:57.038413+0000","last_undegraded":"2024-03-18T15:58:57.038413+0000","last_fullsized":"2024-03-18T15:58:57.038413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:18.430632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038586+0000","last_change":"2024-03-18T15:57:56.176784+0000","last_active":"2024-03-18T15:58:57.038586+0000","last_peered":"2024-03-18T15:58:57.038586+0000","last_clean":"2024-03-18T15:58:57.038586+0000","last_became_active":"2024-03-18T15:57:56.176468+0000","last_became_peered":"2024-03-18T15:57:56.176468+0000","last_unstale":"2024-03-18T15:58:57.038586+0000","last_undegraded":"2024-03-18T15:58:57.038586+0000","last_fullsized":"2024-03-18T15:58:57.038586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:50.941724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820719+0000","last_change":"2024-03-18T15:57:57.183492+0000","last_active":"2024-03-18T15:58:56.820719+0000","last_peered":"2024-03-18T15:58:56.820719+0000","last_clean":"2024-03-18T15:58:56.820719+0000","last_became_active":"2024-03-18T15:57:57.183156+0000","last_became_peered":"2024-03-18T15:57:57.183156+0000","last_unstale":"2024-03-18T15:58:56.820719+0000","last_undegraded":"2024-03-18T15:58:56.820719+0000","last_fullsized":"2024-03-18T15:58:56.820719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:41.441788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.582136+0000","last_change":"2024-03-18T15:58:56.581740+0000","last_active":"2024-03-18T15:58:56.582136+0000","last_peered":"2024-03-18T15:58:56.582136+0000","last_clean":"2024-03-18T15:58:56.582136+0000","last_became_active":"2024-03-18T15:58:56.573608+0000","last_became_peered":"2024-03-18T15:58:56.573608+0000","last_unstale":"2024-03-18T15:58:56.582136+0000","last_undegraded":"2024-03-18T15:58:56.582136+0000","last_fullsized":"2024-03-18T15:58:56.582136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:59.170341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572372+0000","last_change":"2024-03-18T15:57:57.180588+0000","last_active":"2024-03-18T15:58:56.572372+0000","last_peered":"2024-03-18T15:58:56.572372+0000","last_clean":"2024-03-18T15:58:56.572372+0000","last_became_active":"2024-03-18T15:57:57.180245+0000","last_became_peered":"2024-03-18T15:57:57.180245+0000","last_unstale":"2024-03-18T15:58:56.572372+0000","last_undegraded":"2024-03-18T15:58:56.572372+0000","last_fullsized":"2024-03-18T15:58:56.572372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:38.095406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.575197+0000","last_change":"2024-03-18T15:57:56.179815+0000","last_active":"2024-03-18T15:58:56.575197+0000","last_peered":"2024-03-18T15:58:56.575197+0000","last_clean":"2024-03-18T15:58:56.575197+0000","last_became_active":"2024-03-18T15:57:56.179495+0000","last_became_peered":"2024-03-18T15:57:56.179495+0000","last_unstale":"2024-03-18T15:58:56.575197+0000","last_undegraded":"2024-03-18T15:58:56.575197+0000","last_fullsized":"2024-03-18T15:58:56.575197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:17.876766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573567+0000","last_change":"2024-03-18T15:57:57.180021+0000","last_active":"2024-03-18T15:58:56.573567+0000","last_peered":"2024-03-18T15:58:56.573567+0000","last_clean":"2024-03-18T15:58:56.573567+0000","last_became_active":"2024-03-18T15:57:57.179642+0000","last_became_peered":"2024-03-18T15:57:57.179642+0000","last_unstale":"2024-03-18T15:58:56.573567+0000","last_undegraded":"2024-03-18T15:58:56.573567+0000","last_fullsized":"2024-03-18T15:58:56.573567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:36.573584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:14.277485+0000","last_change":"2024-03-18T15:57:56.178907+0000","last_active":"2024-03-18T15:59:14.277485+0000","last_peered":"2024-03-18T15:59:14.277485+0000","last_clean":"2024-03-18T15:59:14.277485+0000","last_became_active":"2024-03-18T15:57:56.178420+0000","last_became_peered":"2024-03-18T15:57:56.178420+0000","last_unstale":"2024-03-18T15:59:14.277485+0000","last_undegraded":"2024-03-18T15:59:14.277485+0000","last_fullsized":"2024-03-18T15:59:14.277485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:24:45.513178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574129+0000","last_change":"2024-03-18T15:57:57.187726+0000","last_active":"2024-03-18T15:58:56.574129+0000","last_peered":"2024-03-18T15:58:56.574129+0000","last_clean":"2024-03-18T15:58:56.574129+0000","last_became_active":"2024-03-18T15:57:57.187383+0000","last_became_peered":"2024-03-18T15:57:57.187383+0000","last_unstale":"2024-03-18T15:58:56.574129+0000","last_undegraded":"2024-03-18T15:58:56.574129+0000","last_fullsized":"2024-03-18T15:58:56.574129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:23.134839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820636+0000","last_change":"2024-03-18T15:57:56.175120+0000","last_active":"2024-03-18T15:58:56.820636+0000","last_peered":"2024-03-18T15:58:56.820636+0000","last_clean":"2024-03-18T15:58:56.820636+0000","last_became_active":"2024-03-18T15:57:56.174554+0000","last_became_peered":"2024-03-18T15:57:56.174554+0000","last_unstale":"2024-03-18T15:58:56.820636+0000","last_undegraded":"2024-03-18T15:58:56.820636+0000","last_fullsized":"2024-03-18T15:58:56.820636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:16.894313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573231+0000","last_change":"2024-03-18T15:57:57.192612+0000","last_active":"2024-03-18T15:58:56.573231+0000","last_peered":"2024-03-18T15:58:56.573231+0000","last_clean":"2024-03-18T15:58:56.573231+0000","last_became_active":"2024-03-18T15:57:57.192284+0000","last_became_peered":"2024-03-18T15:57:57.192284+0000","last_unstale":"2024-03-18T15:58:56.573231+0000","last_undegraded":"2024-03-18T15:58:56.573231+0000","last_fullsized":"2024-03-18T15:58:56.573231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:53.597362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.580958+0000","last_change":"2024-03-18T15:57:56.165164+0000","last_active":"2024-03-18T15:58:56.580958+0000","last_peered":"2024-03-18T15:58:56.580958+0000","last_clean":"2024-03-18T15:58:56.580958+0000","last_became_active":"2024-03-18T15:57:56.164844+0000","last_became_peered":"2024-03-18T15:57:56.164844+0000","last_unstale":"2024-03-18T15:58:56.580958+0000","last_undegraded":"2024-03-18T15:58:56.580958+0000","last_fullsized":"2024-03-18T15:58:56.580958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:36.754823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038173+0000","last_change":"2024-03-18T15:57:57.180176+0000","last_active":"2024-03-18T15:58:57.038173+0000","last_peered":"2024-03-18T15:58:57.038173+0000","last_clean":"2024-03-18T15:58:57.038173+0000","last_became_active":"2024-03-18T15:57:57.179832+0000","last_became_peered":"2024-03-18T15:57:57.179832+0000","last_unstale":"2024-03-18T15:58:57.038173+0000","last_undegraded":"2024-03-18T15:58:57.038173+0000","last_fullsized":"2024-03-18T15:58:57.038173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:32.249126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573371+0000","last_change":"2024-03-18T15:57:56.175404+0000","last_active":"2024-03-18T15:58:56.573371+0000","last_peered":"2024-03-18T15:58:56.573371+0000","last_clean":"2024-03-18T15:58:56.573371+0000","last_became_active":"2024-03-18T15:57:56.174877+0000","last_became_peered":"2024-03-18T15:57:56.174877+0000","last_unstale":"2024-03-18T15:58:56.573371+0000","last_undegraded":"2024-03-18T15:58:56.573371+0000","last_fullsized":"2024-03-18T15:58:56.573371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:39.536240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.568655+0000","last_change":"2024-03-18T15:57:57.178213+0000","last_active":"2024-03-18T15:58:56.568655+0000","last_peered":"2024-03-18T15:58:56.568655+0000","last_clean":"2024-03-18T15:58:56.568655+0000","last_became_active":"2024-03-18T15:57:57.177863+0000","last_became_peered":"2024-03-18T15:57:57.177863+0000","last_unstale":"2024-03-18T15:58:56.568655+0000","last_undegraded":"2024-03-18T15:58:56.568655+0000","last_fullsized":"2024-03-18T15:58:56.568655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:42.988890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:13.276122+0000","last_change":"2024-03-18T15:57:56.172506+0000","last_active":"2024-03-18T15:59:13.276122+0000","last_peered":"2024-03-18T15:59:13.276122+0000","last_clean":"2024-03-18T15:59:13.276122+0000","last_became_active":"2024-03-18T15:57:56.172153+0000","last_became_peered":"2024-03-18T15:57:56.172153+0000","last_unstale":"2024-03-18T15:59:13.276122+0000","last_undegraded":"2024-03-18T15:59:13.276122+0000","last_fullsized":"2024-03-18T15:59:13.276122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:30.563167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572556+0000","last_change":"2024-03-18T15:57:57.181506+0000","last_active":"2024-03-18T15:58:56.572556+0000","last_peered":"2024-03-18T15:58:56.572556+0000","last_clean":"2024-03-18T15:58:56.572556+0000","last_became_active":"2024-03-18T15:57:57.181139+0000","last_became_peered":"2024-03-18T15:57:57.181139+0000","last_unstale":"2024-03-18T15:58:56.572556+0000","last_undegraded":"2024-03-18T15:58:56.572556+0000","last_fullsized":"2024-03-18T15:58:56.572556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:40.239719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.771996+0000","last_change":"2024-03-18T15:58:56.771996+0000","last_active":"2024-03-18T15:58:56.771996+0000","last_peered":"2024-03-18T15:58:56.771996+0000","last_clean":"2024-03-18T15:58:56.771996+0000","last_became_active":"2024-03-18T15:58:56.769351+0000","last_became_peered":"2024-03-18T15:58:56.769351+0000","last_unstale":"2024-03-18T15:58:56.771996+0000","last_undegraded":"2024-03-18T15:58:56.771996+0000","last_fullsized":"2024-03-18T15:58:56.771996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:00.300021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820715+0000","last_change":"2024-03-18T15:57:57.186766+0000","last_active":"2024-03-18T15:58:56.820715+0000","last_peered":"2024-03-18T15:58:56.820715+0000","last_clean":"2024-03-18T15:58:56.820715+0000","last_became_active":"2024-03-18T15:57:57.186453+0000","last_became_peered":"2024-03-18T15:57:57.186453+0000","last_unstale":"2024-03-18T15:58:56.820715+0000","last_undegraded":"2024-03-18T15:58:56.820715+0000","last_fullsized":"2024-03-18T15:58:56.820715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:19.716635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573672+0000","last_change":"2024-03-18T15:57:56.174536+0000","last_active":"2024-03-18T15:58:56.573672+0000","last_peered":"2024-03-18T15:58:56.573672+0000","last_clean":"2024-03-18T15:58:56.573672+0000","last_became_active":"2024-03-18T15:57:56.174216+0000","last_became_peered":"2024-03-18T15:57:56.174216+0000","last_unstale":"2024-03-18T15:58:56.573672+0000","last_undegraded":"2024-03-18T15:58:56.573672+0000","last_fullsized":"2024-03-18T15:58:56.573672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:50.324174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572582+0000","last_change":"2024-03-18T15:57:57.178511+0000","last_active":"2024-03-18T15:58:56.572582+0000","last_peered":"2024-03-18T15:58:56.572582+0000","last_clean":"2024-03-18T15:58:56.572582+0000","last_became_active":"2024-03-18T15:57:57.178117+0000","last_became_peered":"2024-03-18T15:57:57.178117+0000","last_unstale":"2024-03-18T15:58:56.572582+0000","last_undegraded":"2024-03-18T15:58:56.572582+0000","last_fullsized":"2024-03-18T15:58:56.572582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:52.891086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'30","reported_seq":205,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:14.277103+0000","last_change":"2024-03-18T15:57:56.173372+0000","last_active":"2024-03-18T15:59:14.277103+0000","last_peered":"2024-03-18T15:59:14.277103+0000","last_clean":"2024-03-18T15:59:14.277103+0000","last_became_active":"2024-03-18T15:57:56.173049+0000","last_became_peered":"2024-03-18T15:57:56.173049+0000","last_unstale":"2024-03-18T15:59:14.277103+0000","last_undegraded":"2024-03-18T15:59:14.277103+0000","last_fullsized":"2024-03-18T15:59:14.277103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:01.655747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60023,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":156,"num_read_kb":135,"num_write":31,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574763+0000","last_change":"2024-03-18T15:57:57.183725+0000","last_active":"2024-03-18T15:58:56.574763+0000","last_peered":"2024-03-18T15:58:56.574763+0000","last_clean":"2024-03-18T15:58:56.574763+0000","last_became_active":"2024-03-18T15:57:57.183391+0000","last_became_peered":"2024-03-18T15:57:57.183391+0000","last_unstale":"2024-03-18T15:58:56.574763+0000","last_undegraded":"2024-03-18T15:58:56.574763+0000","last_fullsized":"2024-03-18T15:58:56.574763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:42.945519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572633+0000","last_change":"2024-03-18T15:57:56.179534+0000","last_active":"2024-03-18T15:58:56.572633+0000","last_peered":"2024-03-18T15:58:56.572633+0000","last_clean":"2024-03-18T15:58:56.572633+0000","last_became_active":"2024-03-18T15:57:56.179217+0000","last_became_peered":"2024-03-18T15:57:56.179217+0000","last_unstale":"2024-03-18T15:58:56.572633+0000","last_undegraded":"2024-03-18T15:58:56.572633+0000","last_fullsized":"2024-03-18T15:58:56.572633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:19.176423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581245+0000","last_change":"2024-03-18T15:57:57.188958+0000","last_active":"2024-03-18T15:58:56.581245+0000","last_peered":"2024-03-18T15:58:56.581245+0000","last_clean":"2024-03-18T15:58:56.581245+0000","last_became_active":"2024-03-18T15:57:57.188586+0000","last_became_peered":"2024-03-18T15:57:57.188586+0000","last_unstale":"2024-03-18T15:58:56.581245+0000","last_undegraded":"2024-03-18T15:58:56.581245+0000","last_fullsized":"2024-03-18T15:58:56.581245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:48.122539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572531+0000","last_change":"2024-03-18T15:57:56.180530+0000","last_active":"2024-03-18T15:58:56.572531+0000","last_peered":"2024-03-18T15:58:56.572531+0000","last_clean":"2024-03-18T15:58:56.572531+0000","last_became_active":"2024-03-18T15:57:56.180019+0000","last_became_peered":"2024-03-18T15:57:56.180019+0000","last_unstale":"2024-03-18T15:58:56.572531+0000","last_undegraded":"2024-03-18T15:58:56.572531+0000","last_fullsized":"2024-03-18T15:58:56.572531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:04.119730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581996+0000","last_change":"2024-03-18T15:58:56.581996+0000","last_active":"2024-03-18T15:58:56.581996+0000","last_peered":"2024-03-18T15:58:56.581996+0000","last_clean":"2024-03-18T15:58:56.581996+0000","last_became_active":"2024-03-18T15:58:56.581728+0000","last_became_peered":"2024-03-18T15:58:56.581728+0000","last_unstale":"2024-03-18T15:58:56.581996+0000","last_undegraded":"2024-03-18T15:58:56.581996+0000","last_fullsized":"2024-03-18T15:58:56.581996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:25.718628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":99,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:14.287222+0000","last_change":"2024-03-18T15:57:56.172587+0000","last_active":"2024-03-18T15:59:14.287222+0000","last_peered":"2024-03-18T15:59:14.287222+0000","last_clean":"2024-03-18T15:59:14.287222+0000","last_became_active":"2024-03-18T15:57:56.172246+0000","last_became_peered":"2024-03-18T15:57:56.172246+0000","last_unstale":"2024-03-18T15:59:14.287222+0000","last_undegraded":"2024-03-18T15:59:14.287222+0000","last_fullsized":"2024-03-18T15:59:14.287222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.543059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569269+0000","last_change":"2024-03-18T15:57:57.181899+0000","last_active":"2024-03-18T15:58:56.569269+0000","last_peered":"2024-03-18T15:58:56.569269+0000","last_clean":"2024-03-18T15:58:56.569269+0000","last_became_active":"2024-03-18T15:57:57.181559+0000","last_became_peered":"2024-03-18T15:57:57.181559+0000","last_unstale":"2024-03-18T15:58:56.569269+0000","last_undegraded":"2024-03-18T15:58:56.569269+0000","last_fullsized":"2024-03-18T15:58:56.569269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:17:17.504153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573770+0000","last_change":"2024-03-18T15:57:56.179375+0000","last_active":"2024-03-18T15:58:56.573770+0000","last_peered":"2024-03-18T15:58:56.573770+0000","last_clean":"2024-03-18T15:58:56.573770+0000","last_became_active":"2024-03-18T15:57:56.178990+0000","last_became_peered":"2024-03-18T15:57:56.178990+0000","last_unstale":"2024-03-18T15:58:56.573770+0000","last_undegraded":"2024-03-18T15:58:56.573770+0000","last_fullsized":"2024-03-18T15:58:56.573770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:36.181030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572251+0000","last_change":"2024-03-18T15:57:57.188135+0000","last_active":"2024-03-18T15:58:56.572251+0000","last_peered":"2024-03-18T15:58:56.572251+0000","last_clean":"2024-03-18T15:58:56.572251+0000","last_became_active":"2024-03-18T15:57:57.187811+0000","last_became_peered":"2024-03-18T15:57:57.187811+0000","last_unstale":"2024-03-18T15:58:56.572251+0000","last_undegraded":"2024-03-18T15:58:56.572251+0000","last_fullsized":"2024-03-18T15:58:56.572251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:11.235420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574600+0000","last_change":"2024-03-18T15:57:56.181774+0000","last_active":"2024-03-18T15:58:56.574600+0000","last_peered":"2024-03-18T15:58:56.574600+0000","last_clean":"2024-03-18T15:58:56.574600+0000","last_became_active":"2024-03-18T15:57:56.181360+0000","last_became_peered":"2024-03-18T15:57:56.181360+0000","last_unstale":"2024-03-18T15:58:56.574600+0000","last_undegraded":"2024-03-18T15:58:56.574600+0000","last_fullsized":"2024-03-18T15:58:56.574600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:08.367151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570042+0000","last_change":"2024-03-18T15:57:57.188960+0000","last_active":"2024-03-18T15:58:56.570042+0000","last_peered":"2024-03-18T15:58:56.570042+0000","last_clean":"2024-03-18T15:58:56.570042+0000","last_became_active":"2024-03-18T15:57:57.187528+0000","last_became_peered":"2024-03-18T15:57:57.187528+0000","last_unstale":"2024-03-18T15:58:56.570042+0000","last_undegraded":"2024-03-18T15:58:56.570042+0000","last_fullsized":"2024-03-18T15:58:56.570042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:29.259977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.821606+0000","last_change":"2024-03-18T15:58:56.821606+0000","last_active":"2024-03-18T15:58:56.821606+0000","last_peered":"2024-03-18T15:58:56.821606+0000","last_clean":"2024-03-18T15:58:56.821606+0000","last_became_active":"2024-03-18T15:58:56.821283+0000","last_became_peered":"2024-03-18T15:58:56.821283+0000","last_unstale":"2024-03-18T15:58:56.821606+0000","last_undegraded":"2024-03-18T15:58:56.821606+0000","last_fullsized":"2024-03-18T15:58:56.821606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:02.895290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820973+0000","last_change":"2024-03-18T15:57:57.189238+0000","last_active":"2024-03-18T15:58:56.820973+0000","last_peered":"2024-03-18T15:58:56.820973+0000","last_clean":"2024-03-18T15:58:56.820973+0000","last_became_active":"2024-03-18T15:57:57.188900+0000","last_became_peered":"2024-03-18T15:57:57.188900+0000","last_unstale":"2024-03-18T15:58:56.820973+0000","last_undegraded":"2024-03-18T15:58:56.820973+0000","last_fullsized":"2024-03-18T15:58:56.820973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:16.810976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569598+0000","last_change":"2024-03-18T15:57:56.178765+0000","last_active":"2024-03-18T15:58:56.569598+0000","last_peered":"2024-03-18T15:58:56.569598+0000","last_clean":"2024-03-18T15:58:56.569598+0000","last_became_active":"2024-03-18T15:57:56.178399+0000","last_became_peered":"2024-03-18T15:57:56.178399+0000","last_unstale":"2024-03-18T15:58:56.569598+0000","last_undegraded":"2024-03-18T15:58:56.569598+0000","last_fullsized":"2024-03-18T15:58:56.569598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:46:07.844453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820928+0000","last_change":"2024-03-18T15:57:57.173968+0000","last_active":"2024-03-18T15:58:56.820928+0000","last_peered":"2024-03-18T15:58:56.820928+0000","last_clean":"2024-03-18T15:58:56.820928+0000","last_became_active":"2024-03-18T15:57:57.173605+0000","last_became_peered":"2024-03-18T15:57:57.173605+0000","last_unstale":"2024-03-18T15:58:56.820928+0000","last_undegraded":"2024-03-18T15:58:56.820928+0000","last_fullsized":"2024-03-18T15:58:56.820928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:34.676393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573566+0000","last_change":"2024-03-18T15:57:57.186561+0000","last_active":"2024-03-18T15:58:56.573566+0000","last_peered":"2024-03-18T15:58:56.573566+0000","last_clean":"2024-03-18T15:58:56.573566+0000","last_became_active":"2024-03-18T15:57:57.186260+0000","last_became_peered":"2024-03-18T15:57:57.186260+0000","last_unstale":"2024-03-18T15:58:56.573566+0000","last_undegraded":"2024-03-18T15:58:56.573566+0000","last_fullsized":"2024-03-18T15:58:56.573566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:54:06.646480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":99,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:59:15.287836+0000","last_change":"2024-03-18T15:57:56.164175+0000","last_active":"2024-03-18T15:59:15.287836+0000","last_peered":"2024-03-18T15:59:15.287836+0000","last_clean":"2024-03-18T15:59:15.287836+0000","last_became_active":"2024-03-18T15:57:56.163724+0000","last_became_peered":"2024-03-18T15:57:56.163724+0000","last_unstale":"2024-03-18T15:59:15.287836+0000","last_undegraded":"2024-03-18T15:59:15.287836+0000","last_fullsized":"2024-03-18T15:59:15.287836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:48:53.596211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038203+0000","last_change":"2024-03-18T15:57:57.191507+0000","last_active":"2024-03-18T15:58:57.038203+0000","last_peered":"2024-03-18T15:58:57.038203+0000","last_clean":"2024-03-18T15:58:57.038203+0000","last_became_active":"2024-03-18T15:57:57.191182+0000","last_became_peered":"2024-03-18T15:57:57.191182+0000","last_unstale":"2024-03-18T15:58:57.038203+0000","last_undegraded":"2024-03-18T15:58:57.038203+0000","last_fullsized":"2024-03-18T15:58:57.038203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:39.878675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768679+0000","last_change":"2024-03-18T15:57:56.169490+0000","last_active":"2024-03-18T15:58:56.768679+0000","last_peered":"2024-03-18T15:58:56.768679+0000","last_clean":"2024-03-18T15:58:56.768679+0000","last_became_active":"2024-03-18T15:57:56.169107+0000","last_became_peered":"2024-03-18T15:57:56.169107+0000","last_unstale":"2024-03-18T15:58:56.768679+0000","last_undegraded":"2024-03-18T15:58:56.768679+0000","last_fullsized":"2024-03-18T15:58:56.768679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:59.722909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569340+0000","last_change":"2024-03-18T15:56:43.270716+0000","last_active":"2024-03-18T15:58:56.569340+0000","last_peered":"2024-03-18T15:58:56.569340+0000","last_clean":"2024-03-18T15:58:56.569340+0000","last_became_active":"2024-03-18T15:56:43.270316+0000","last_became_peered":"2024-03-18T15:56:43.270316+0000","last_unstale":"2024-03-18T15:58:56.569340+0000","last_undegraded":"2024-03-18T15:58:56.569340+0000","last_fullsized":"2024-03-18T15:58:56.569340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:53:17.916048+0000","last_clean_scrub_stamp":"2024-03-18T15:53:17.916048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:00.779948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.580856+0000","last_change":"2024-03-18T15:57:57.172265+0000","last_active":"2024-03-18T15:58:56.580856+0000","last_peered":"2024-03-18T15:58:56.580856+0000","last_clean":"2024-03-18T15:58:56.580856+0000","last_became_active":"2024-03-18T15:57:57.171962+0000","last_became_peered":"2024-03-18T15:57:57.171962+0000","last_unstale":"2024-03-18T15:58:56.580856+0000","last_undegraded":"2024-03-18T15:58:56.580856+0000","last_fullsized":"2024-03-18T15:58:56.580856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:41:44.402012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.577920+0000","last_change":"2024-03-18T15:58:56.577920+0000","last_active":"2024-03-18T15:58:56.577920+0000","last_peered":"2024-03-18T15:58:56.577920+0000","last_clean":"2024-03-18T15:58:56.577920+0000","last_became_active":"2024-03-18T15:58:56.574742+0000","last_became_peered":"2024-03-18T15:58:56.574742+0000","last_unstale":"2024-03-18T15:58:56.577920+0000","last_undegraded":"2024-03-18T15:58:56.577920+0000","last_fullsized":"2024-03-18T15:58:56.577920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:00.046050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574865+0000","last_change":"2024-03-18T15:57:56.178710+0000","last_active":"2024-03-18T15:58:56.574865+0000","last_peered":"2024-03-18T15:58:56.574865+0000","last_clean":"2024-03-18T15:58:56.574865+0000","last_became_active":"2024-03-18T15:57:56.178388+0000","last_became_peered":"2024-03-18T15:57:56.178388+0000","last_unstale":"2024-03-18T15:58:56.574865+0000","last_undegraded":"2024-03-18T15:58:56.574865+0000","last_fullsized":"2024-03-18T15:58:56.574865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:23.093812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581151+0000","last_change":"2024-03-18T15:57:57.168951+0000","last_active":"2024-03-18T15:58:56.581151+0000","last_peered":"2024-03-18T15:58:56.581151+0000","last_clean":"2024-03-18T15:58:56.581151+0000","last_became_active":"2024-03-18T15:57:57.168567+0000","last_became_peered":"2024-03-18T15:57:57.168567+0000","last_unstale":"2024-03-18T15:58:56.581151+0000","last_undegraded":"2024-03-18T15:58:56.581151+0000","last_fullsized":"2024-03-18T15:58:56.581151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:41.171254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569059+0000","last_change":"2024-03-18T15:57:56.171665+0000","last_active":"2024-03-18T15:58:56.569059+0000","last_peered":"2024-03-18T15:58:56.569059+0000","last_clean":"2024-03-18T15:58:56.569059+0000","last_became_active":"2024-03-18T15:57:56.171282+0000","last_became_peered":"2024-03-18T15:57:56.171282+0000","last_unstale":"2024-03-18T15:58:56.569059+0000","last_undegraded":"2024-03-18T15:58:56.569059+0000","last_fullsized":"2024-03-18T15:58:56.569059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:03:27.523561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038327+0000","last_change":"2024-03-18T15:57:57.185647+0000","last_active":"2024-03-18T15:58:57.038327+0000","last_peered":"2024-03-18T15:58:57.038327+0000","last_clean":"2024-03-18T15:58:57.038327+0000","last_became_active":"2024-03-18T15:57:57.185352+0000","last_became_peered":"2024-03-18T15:57:57.185352+0000","last_unstale":"2024-03-18T15:58:57.038327+0000","last_undegraded":"2024-03-18T15:58:57.038327+0000","last_fullsized":"2024-03-18T15:58:57.038327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:38.247776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.768809+0000","last_change":"2024-03-18T15:57:56.170878+0000","last_active":"2024-03-18T15:58:56.768809+0000","last_peered":"2024-03-18T15:58:56.768809+0000","last_clean":"2024-03-18T15:58:56.768809+0000","last_became_active":"2024-03-18T15:57:56.170558+0000","last_became_peered":"2024-03-18T15:57:56.170558+0000","last_unstale":"2024-03-18T15:58:56.768809+0000","last_undegraded":"2024-03-18T15:58:56.768809+0000","last_fullsized":"2024-03-18T15:58:56.768809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:26:39.963384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820621+0000","last_change":"2024-03-18T15:57:57.181012+0000","last_active":"2024-03-18T15:58:56.820621+0000","last_peered":"2024-03-18T15:58:56.820621+0000","last_clean":"2024-03-18T15:58:56.820621+0000","last_became_active":"2024-03-18T15:57:57.180613+0000","last_became_peered":"2024-03-18T15:57:57.180613+0000","last_unstale":"2024-03-18T15:58:56.820621+0000","last_undegraded":"2024-03-18T15:58:56.820621+0000","last_fullsized":"2024-03-18T15:58:56.820621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:22.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572378+0000","last_change":"2024-03-18T15:57:56.171929+0000","last_active":"2024-03-18T15:58:56.572378+0000","last_peered":"2024-03-18T15:58:56.572378+0000","last_clean":"2024-03-18T15:58:56.572378+0000","last_became_active":"2024-03-18T15:57:56.171570+0000","last_became_peered":"2024-03-18T15:57:56.171570+0000","last_unstale":"2024-03-18T15:58:56.572378+0000","last_undegraded":"2024-03-18T15:58:56.572378+0000","last_fullsized":"2024-03-18T15:58:56.572378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:11.579956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573376+0000","last_change":"2024-03-18T15:57:57.178118+0000","last_active":"2024-03-18T15:58:56.573376+0000","last_peered":"2024-03-18T15:58:56.573376+0000","last_clean":"2024-03-18T15:58:56.573376+0000","last_became_active":"2024-03-18T15:57:57.177764+0000","last_became_peered":"2024-03-18T15:57:57.177764+0000","last_unstale":"2024-03-18T15:58:56.573376+0000","last_undegraded":"2024-03-18T15:58:56.573376+0000","last_fullsized":"2024-03-18T15:58:56.573376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:07.519544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038062+0000","last_change":"2024-03-18T15:57:56.168485+0000","last_active":"2024-03-18T15:58:57.038062+0000","last_peered":"2024-03-18T15:58:57.038062+0000","last_clean":"2024-03-18T15:58:57.038062+0000","last_became_active":"2024-03-18T15:57:56.168124+0000","last_became_peered":"2024-03-18T15:57:56.168124+0000","last_unstale":"2024-03-18T15:58:57.038062+0000","last_undegraded":"2024-03-18T15:58:57.038062+0000","last_fullsized":"2024-03-18T15:58:57.038062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:09.465975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569757+0000","last_change":"2024-03-18T15:57:57.184781+0000","last_active":"2024-03-18T15:58:56.569757+0000","last_peered":"2024-03-18T15:58:56.569757+0000","last_clean":"2024-03-18T15:58:56.569757+0000","last_became_active":"2024-03-18T15:57:57.184431+0000","last_became_peered":"2024-03-18T15:57:57.184431+0000","last_unstale":"2024-03-18T15:58:56.569757+0000","last_undegraded":"2024-03-18T15:58:56.569757+0000","last_fullsized":"2024-03-18T15:58:56.569757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:32.945288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572620+0000","last_change":"2024-03-18T15:57:56.180295+0000","last_active":"2024-03-18T15:58:56.572620+0000","last_peered":"2024-03-18T15:58:56.572620+0000","last_clean":"2024-03-18T15:58:56.572620+0000","last_became_active":"2024-03-18T15:57:56.179956+0000","last_became_peered":"2024-03-18T15:57:56.179956+0000","last_unstale":"2024-03-18T15:58:56.572620+0000","last_undegraded":"2024-03-18T15:58:56.572620+0000","last_fullsized":"2024-03-18T15:58:56.572620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:27:49.664982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038445+0000","last_change":"2024-03-18T15:57:57.187421+0000","last_active":"2024-03-18T15:58:57.038445+0000","last_peered":"2024-03-18T15:58:57.038445+0000","last_clean":"2024-03-18T15:58:57.038445+0000","last_became_active":"2024-03-18T15:57:57.185978+0000","last_became_peered":"2024-03-18T15:57:57.185978+0000","last_unstale":"2024-03-18T15:58:57.038445+0000","last_undegraded":"2024-03-18T15:58:57.038445+0000","last_fullsized":"2024-03-18T15:58:57.038445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:07:59.246048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.574336+0000","last_change":"2024-03-18T15:57:56.178118+0000","last_active":"2024-03-18T15:58:56.574336+0000","last_peered":"2024-03-18T15:58:56.574336+0000","last_clean":"2024-03-18T15:58:56.574336+0000","last_became_active":"2024-03-18T15:57:56.177758+0000","last_became_peered":"2024-03-18T15:57:56.177758+0000","last_unstale":"2024-03-18T15:58:56.574336+0000","last_undegraded":"2024-03-18T15:58:56.574336+0000","last_fullsized":"2024-03-18T15:58:56.574336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:34.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570140+0000","last_change":"2024-03-18T15:57:57.182666+0000","last_active":"2024-03-18T15:58:56.570140+0000","last_peered":"2024-03-18T15:58:56.570140+0000","last_clean":"2024-03-18T15:58:56.570140+0000","last_became_active":"2024-03-18T15:57:57.182328+0000","last_became_peered":"2024-03-18T15:57:57.182328+0000","last_unstale":"2024-03-18T15:58:56.570140+0000","last_undegraded":"2024-03-18T15:58:56.570140+0000","last_fullsized":"2024-03-18T15:58:56.570140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:55.132441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573040+0000","last_change":"2024-03-18T15:57:56.177101+0000","last_active":"2024-03-18T15:58:56.573040+0000","last_peered":"2024-03-18T15:58:56.573040+0000","last_clean":"2024-03-18T15:58:56.573040+0000","last_became_active":"2024-03-18T15:57:56.176734+0000","last_became_peered":"2024-03-18T15:57:56.176734+0000","last_unstale":"2024-03-18T15:58:56.573040+0000","last_undegraded":"2024-03-18T15:58:56.573040+0000","last_fullsized":"2024-03-18T15:58:56.573040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:55.406210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569787+0000","last_change":"2024-03-18T15:57:57.183892+0000","last_active":"2024-03-18T15:58:56.569787+0000","last_peered":"2024-03-18T15:58:56.569787+0000","last_clean":"2024-03-18T15:58:56.569787+0000","last_became_active":"2024-03-18T15:57:57.183121+0000","last_became_peered":"2024-03-18T15:57:57.183121+0000","last_unstale":"2024-03-18T15:58:56.569787+0000","last_undegraded":"2024-03-18T15:58:56.569787+0000","last_fullsized":"2024-03-18T15:58:56.569787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:11:13.504169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573457+0000","last_change":"2024-03-18T15:57:56.172479+0000","last_active":"2024-03-18T15:58:56.573457+0000","last_peered":"2024-03-18T15:58:56.573457+0000","last_clean":"2024-03-18T15:58:56.573457+0000","last_became_active":"2024-03-18T15:57:56.172120+0000","last_became_peered":"2024-03-18T15:57:56.172120+0000","last_unstale":"2024-03-18T15:58:56.573457+0000","last_undegraded":"2024-03-18T15:58:56.573457+0000","last_fullsized":"2024-03-18T15:58:56.573457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:15.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569645+0000","last_change":"2024-03-18T15:57:57.183674+0000","last_active":"2024-03-18T15:58:56.569645+0000","last_peered":"2024-03-18T15:58:56.569645+0000","last_clean":"2024-03-18T15:58:56.569645+0000","last_became_active":"2024-03-18T15:57:57.183270+0000","last_became_peered":"2024-03-18T15:57:57.183270+0000","last_unstale":"2024-03-18T15:58:56.569645+0000","last_undegraded":"2024-03-18T15:58:56.569645+0000","last_fullsized":"2024-03-18T15:58:56.569645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:43.258995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581093+0000","last_change":"2024-03-18T15:57:56.174497+0000","last_active":"2024-03-18T15:58:56.581093+0000","last_peered":"2024-03-18T15:58:56.581093+0000","last_clean":"2024-03-18T15:58:56.581093+0000","last_became_active":"2024-03-18T15:57:56.173862+0000","last_became_peered":"2024-03-18T15:57:56.173862+0000","last_unstale":"2024-03-18T15:58:56.581093+0000","last_undegraded":"2024-03-18T15:58:56.581093+0000","last_fullsized":"2024-03-18T15:58:56.581093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:55.143227+0000","last_clean_scrub_stamp":"2024-03-18T15:57:55.143227+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:00.362276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569561+0000","last_change":"2024-03-18T15:57:57.179510+0000","last_active":"2024-03-18T15:58:56.569561+0000","last_peered":"2024-03-18T15:58:56.569561+0000","last_clean":"2024-03-18T15:58:56.569561+0000","last_became_active":"2024-03-18T15:57:57.179178+0000","last_became_peered":"2024-03-18T15:57:57.179178+0000","last_unstale":"2024-03-18T15:58:56.569561+0000","last_undegraded":"2024-03-18T15:58:56.569561+0000","last_fullsized":"2024-03-18T15:58:56.569561+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:46.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.569814+0000","last_change":"2024-03-18T15:57:57.186842+0000","last_active":"2024-03-18T15:58:56.569814+0000","last_peered":"2024-03-18T15:58:56.569814+0000","last_clean":"2024-03-18T15:58:56.569814+0000","last_became_active":"2024-03-18T15:57:57.186501+0000","last_became_peered":"2024-03-18T15:57:57.186501+0000","last_unstale":"2024-03-18T15:58:56.569814+0000","last_undegraded":"2024-03-18T15:58:56.569814+0000","last_fullsized":"2024-03-18T15:58:56.569814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:24:11.988494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572387+0000","last_change":"2024-03-18T15:57:57.177630+0000","last_active":"2024-03-18T15:58:56.572387+0000","last_peered":"2024-03-18T15:58:56.572387+0000","last_clean":"2024-03-18T15:58:56.572387+0000","last_became_active":"2024-03-18T15:57:57.177227+0000","last_became_peered":"2024-03-18T15:57:57.177227+0000","last_unstale":"2024-03-18T15:58:56.572387+0000","last_undegraded":"2024-03-18T15:58:56.572387+0000","last_fullsized":"2024-03-18T15:58:56.572387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:30:27.004129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572543+0000","last_change":"2024-03-18T15:57:57.189048+0000","last_active":"2024-03-18T15:58:56.572543+0000","last_peered":"2024-03-18T15:58:56.572543+0000","last_clean":"2024-03-18T15:58:56.572543+0000","last_became_active":"2024-03-18T15:57:57.188704+0000","last_became_peered":"2024-03-18T15:57:57.188704+0000","last_unstale":"2024-03-18T15:58:56.572543+0000","last_undegraded":"2024-03-18T15:58:56.572543+0000","last_fullsized":"2024-03-18T15:58:56.572543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:10:53.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.820482+0000","last_change":"2024-03-18T15:57:57.178611+0000","last_active":"2024-03-18T15:58:56.820482+0000","last_peered":"2024-03-18T15:58:56.820482+0000","last_clean":"2024-03-18T15:58:56.820482+0000","last_became_active":"2024-03-18T15:57:57.178305+0000","last_became_peered":"2024-03-18T15:57:57.178305+0000","last_unstale":"2024-03-18T15:58:56.820482+0000","last_undegraded":"2024-03-18T15:58:56.820482+0000","last_fullsized":"2024-03-18T15:58:56.820482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:53:51.925278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581087+0000","last_change":"2024-03-18T15:57:57.177928+0000","last_active":"2024-03-18T15:58:56.581087+0000","last_peered":"2024-03-18T15:58:56.581087+0000","last_clean":"2024-03-18T15:58:56.581087+0000","last_became_active":"2024-03-18T15:57:57.177364+0000","last_became_peered":"2024-03-18T15:57:57.177364+0000","last_unstale":"2024-03-18T15:58:56.581087+0000","last_undegraded":"2024-03-18T15:58:56.581087+0000","last_fullsized":"2024-03-18T15:58:56.581087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:59.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.570620+0000","last_change":"2024-03-18T15:57:57.188080+0000","last_active":"2024-03-18T15:58:56.570620+0000","last_peered":"2024-03-18T15:58:56.570620+0000","last_clean":"2024-03-18T15:58:56.570620+0000","last_became_active":"2024-03-18T15:57:57.187767+0000","last_became_peered":"2024-03-18T15:57:57.187767+0000","last_unstale":"2024-03-18T15:58:56.570620+0000","last_undegraded":"2024-03-18T15:58:56.570620+0000","last_fullsized":"2024-03-18T15:58:56.570620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:26.290262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572953+0000","last_change":"2024-03-18T15:57:57.178581+0000","last_active":"2024-03-18T15:58:56.572953+0000","last_peered":"2024-03-18T15:58:56.572953+0000","last_clean":"2024-03-18T15:58:56.572953+0000","last_became_active":"2024-03-18T15:57:57.178168+0000","last_became_peered":"2024-03-18T15:57:57.178168+0000","last_unstale":"2024-03-18T15:58:56.572953+0000","last_undegraded":"2024-03-18T15:58:56.572953+0000","last_fullsized":"2024-03-18T15:58:56.572953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:07:07.601959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038309+0000","last_change":"2024-03-18T15:57:57.188709+0000","last_active":"2024-03-18T15:58:57.038309+0000","last_peered":"2024-03-18T15:58:57.038309+0000","last_clean":"2024-03-18T15:58:57.038309+0000","last_became_active":"2024-03-18T15:57:57.187884+0000","last_became_peered":"2024-03-18T15:57:57.187884+0000","last_unstale":"2024-03-18T15:58:57.038309+0000","last_undegraded":"2024-03-18T15:58:57.038309+0000","last_fullsized":"2024-03-18T15:58:57.038309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:19.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.573475+0000","last_change":"2024-03-18T15:57:57.182489+0000","last_active":"2024-03-18T15:58:56.573475+0000","last_peered":"2024-03-18T15:58:56.573475+0000","last_clean":"2024-03-18T15:58:56.573475+0000","last_became_active":"2024-03-18T15:57:57.176462+0000","last_became_peered":"2024-03-18T15:57:57.176462+0000","last_unstale":"2024-03-18T15:58:56.573475+0000","last_undegraded":"2024-03-18T15:58:56.573475+0000","last_fullsized":"2024-03-18T15:58:56.573475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:46.138422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:57.038009+0000","last_change":"2024-03-18T15:57:57.185848+0000","last_active":"2024-03-18T15:58:57.038009+0000","last_peered":"2024-03-18T15:58:57.038009+0000","last_clean":"2024-03-18T15:58:57.038009+0000","last_became_active":"2024-03-18T15:57:57.185507+0000","last_became_peered":"2024-03-18T15:57:57.185507+0000","last_unstale":"2024-03-18T15:58:57.038009+0000","last_undegraded":"2024-03-18T15:58:57.038009+0000","last_fullsized":"2024-03-18T15:58:57.038009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:09.319325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.572353+0000","last_change":"2024-03-18T15:57:57.189222+0000","last_active":"2024-03-18T15:58:56.572353+0000","last_peered":"2024-03-18T15:58:56.572353+0000","last_clean":"2024-03-18T15:58:56.572353+0000","last_became_active":"2024-03-18T15:57:57.188847+0000","last_became_peered":"2024-03-18T15:57:57.188847+0000","last_unstale":"2024-03-18T15:58:56.572353+0000","last_undegraded":"2024-03-18T15:58:56.572353+0000","last_fullsized":"2024-03-18T15:58:56.572353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:06:15.935552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.581304+0000","last_change":"2024-03-18T15:57:57.183269+0000","last_active":"2024-03-18T15:58:56.581304+0000","last_peered":"2024-03-18T15:58:56.581304+0000","last_clean":"2024-03-18T15:58:56.581304+0000","last_became_active":"2024-03-18T15:57:57.182866+0000","last_became_peered":"2024-03-18T15:57:57.182866+0000","last_unstale":"2024-03-18T15:58:56.581304+0000","last_undegraded":"2024-03-18T15:58:56.581304+0000","last_fullsized":"2024-03-18T15:58:56.581304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:15:08.500842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T15:58:56.577407+0000","last_change":"2024-03-18T15:58:56.577407+0000","last_active":"2024-03-18T15:58:56.577407+0000","last_peered":"2024-03-18T15:58:56.577407+0000","last_clean":"2024-03-18T15:58:56.577407+0000","last_became_active":"2024-03-18T15:58:56.577018+0000","last_became_peered":"2024-03-18T15:58:56.577018+0000","last_unstale":"2024-03-18T15:58:56.577407+0000","last_undegraded":"2024-03-18T15:58:56.577407+0000","last_fullsized":"2024-03-18T15:58:56.577407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T15:57:56.149330+0000","last_clean_scrub_stamp":"2024-03-18T15:57:56.149330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:20.731241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285179,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63633,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":474,"num_read_kb":303,"num_write":156,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351006,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":460588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":1112092,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":460824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":460744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":1052283,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462968,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":460213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":520029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":461484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":521379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":460524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":460650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1051967,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34623,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36262,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":60023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T15:59:18.966 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T15:59:18.967 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T15:59:18.967 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T15:59:18.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T15:59:18.967 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:18 smithi191 ceph-mon[36784]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:18 smithi156 ceph-mon[35919]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:19.216 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:19 smithi191 ceph-mon[36784]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:19 smithi156 ceph-mon[35919]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:19 smithi150 ceph-mon[31946]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:20.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 -- 172.21.15.150:0/3180626239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102ce0 msgr2=0x7fd10c10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:20.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 --2- 172.21.15.150:0/3180626239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102ce0 0x7fd10c10f120 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd10400b0d0 tx=0x7fd10400b3e0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 -- 172.21.15.150:0/3180626239 shutdown_connections 2024-03-18T15:59:20.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 --2- 172.21.15.150:0/3180626239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102ce0 0x7fd10c10f120 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 --2- 172.21.15.150:0/3180626239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd10c108300 0x7fd10c1086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 --2- 172.21.15.150:0/3180626239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102320 0x7fd10c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 -- 172.21.15.150:0/3180626239 >> 172.21.15.150:0/3180626239 conn(0x7fd10c079400 msgr2=0x7fd10c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 -- 172.21.15.150:0/3180626239 shutdown_connections 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.897+0000 7fd111ac7700 1 -- 172.21.15.150:0/3180626239 wait complete. 2024-03-18T15:59:20.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 Processor -- start 2024-03-18T15:59:20.900 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:20 smithi150 ceph-mon[31946]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:20.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 -- start start 2024-03-18T15:59:20.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 0x7fd10c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd10c108300 0x7fd10c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c1125a0 con 0x7fd10c102320 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c112420 con 0x7fd10c102ce0 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd111ac7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c112720 con 0x7fd10c108300 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd10affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd10affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:60946/0 (socket says 172.21.15.150:60946) 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.898+0000 7fd10affd700 1 -- 172.21.15.150:0/1696325710 learned_addr learned my addr 172.21.15.150:0/1696325710 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10b7fe700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 0x7fd10c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 -- 172.21.15.150:0/1696325710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd10c108300 msgr2=0x7fd10c075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd10c108300 0x7fd10c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 -- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 msgr2=0x7fd10c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 0x7fd10c0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 -- 172.21.15.150:0/1696325710 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c19cac0 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10b7fe700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 0x7fd10c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd10affd700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9a0 tx=0x7fd10000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10000cbd0 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd10000cd70 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1000187c0 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10c19cdb0 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.899+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10c19d2a0 con 0x7fd10c102ce0 2024-03-18T15:59:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.900+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10c1039b0 con 0x7fd10c102ce0 2024-03-18T15:59:20.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.901+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd100010430 con 0x7fd10c102ce0 2024-03-18T15:59:20.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.902+0000 7fd108ff9700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 0x7fd0f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:20.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.902+0000 7fd10b7fe700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 0x7fd0f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:20.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.902+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd1000cee40 con 0x7fd10c102ce0 2024-03-18T15:59:20.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.902+0000 7fd10b7fe700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 0x7fd0f40a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009c90 tx=0x7fd0fc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:20.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:20.906+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd10009bdb0 con 0x7fd10c102ce0 2024-03-18T15:59:20.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:20 smithi191 ceph-mon[36784]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:21.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:20 smithi156 ceph-mon[35919]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:21.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.146+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd10c02db70 con 0x7fd10c102ce0 2024-03-18T15:59:21.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.147+0000 7fd108ff9700 1 -- 172.21.15.150:0/1696325710 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd1000bb020 con 0x7fd10c102ce0 2024-03-18T15:59:21.148 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:21.148 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T15:59:21.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 msgr2=0x7fd0f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 0x7fd0f40a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009c90 tx=0x7fd0fc009260 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 msgr2=0x7fd10c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9a0 tx=0x7fd10000ed60 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 shutdown_connections 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd10c108300 0x7fd10c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd10c102320 0x7fd10c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd0f40a29a0 0x7fd0f40a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 --2- 172.21.15.150:0/1696325710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd10c102ce0 0x7fd10c074b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 >> 172.21.15.150:0/1696325710 conn(0x7fd10c079400 msgr2=0x7fd10c100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 shutdown_connections 2024-03-18T15:59:21.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.149+0000 7fd111ac7700 1 -- 172.21.15.150:0/1696325710 wait complete. 2024-03-18T15:59:21.804 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:21 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1696325710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:59:21.835 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T15:59:21.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 -- 172.21.15.150:0/1377256704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb8100700 msgr2=0x7faeb8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 --2- 172.21.15.150:0/1377256704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb8100700 0x7faeb8100b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faea8009900 tx=0x7faea802bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 -- 172.21.15.150:0/1377256704 shutdown_connections 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 --2- 172.21.15.150:0/1377256704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb81022b0 0x7faeb8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 --2- 172.21.15.150:0/1377256704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81018f0 0x7faeb8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 --2- 172.21.15.150:0/1377256704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb8100700 0x7faeb8100b20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 -- 172.21.15.150:0/1377256704 >> 172.21.15.150:0/1377256704 conn(0x7faeb80fbc70 msgr2=0x7faeb80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 -- 172.21.15.150:0/1377256704 shutdown_connections 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.945+0000 7faec047e640 1 -- 172.21.15.150:0/1377256704 wait complete. 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 Processor -- start 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 -- start start 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb81018f0 0x7faeb8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81022b0 0x7faeb819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faeb810d4a0 con 0x7faeb8100700 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faeb810d320 con 0x7faeb81022b0 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faec047e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7faeb810d620 con 0x7faeb81018f0 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39386/0 (socket says 172.21.15.150:39386) 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 -- 172.21.15.150:0/4279322873 learned_addr learned my addr 172.21.15.150:0/4279322873 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebda19640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb81018f0 0x7faeb8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebea1b640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81022b0 0x7faeb819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 -- 172.21.15.150:0/4279322873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb81018f0 msgr2=0x7faeb8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb81018f0 0x7faeb8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 -- 172.21.15.150:0/4279322873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81022b0 msgr2=0x7faeb819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81022b0 0x7faeb819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:21.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.946+0000 7faebe21a640 1 -- 172.21.15.150:0/4279322873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb819f700 con 0x7faeb8100700 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faebe21a640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c00 tx=0x7faea802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8041070 con 0x7faeb8100700 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb819f990 con 0x7faeb8100700 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faea8004450 con 0x7faeb8100700 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8034440 con 0x7faeb8100700 2024-03-18T15:59:21.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.947+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb819fed0 con 0x7faeb8100700 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.948+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb8107a20 con 0x7faeb8100700 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.949+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7faea8044070 con 0x7faeb8100700 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.949+0000 7faeaf7fe640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 0x7faea40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.950+0000 7faebda19640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 0x7faea40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.950+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7faea80f0740 con 0x7faeb8100700 2024-03-18T15:59:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.950+0000 7faebda19640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 0x7faea40a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faeb40095f0 tx=0x7faeb4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:21.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:21.953+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faea80bd200 con 0x7faeb8100700 2024-03-18T15:59:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:21 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1696325710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:59:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:21 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1696325710' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T15:59:22.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.130+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faeb802d420 con 0x7faeb8100700 2024-03-18T15:59:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.132+0000 7faeaf7fe640 1 -- 172.21.15.150:0/4279322873 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7faea80c12a0 con 0x7faeb8100700 2024-03-18T15:59:22.134 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:22.135 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:22.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 msgr2=0x7faea40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 0x7faea40a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faeb40095f0 tx=0x7faeb4005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:22.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 msgr2=0x7faeb8197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c00 tx=0x7faea802c640 comp rx=0 tx=0).stop 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 shutdown_connections 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7faea40a2890 0x7faea40a4d50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faeb81022b0 0x7faeb819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faeb81018f0 0x7faeb8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 --2- 172.21.15.150:0/4279322873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faeb8100700 0x7faeb8197ae0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 >> 172.21.15.150:0/4279322873 conn(0x7faeb80fbc70 msgr2=0x7faeb8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 shutdown_connections 2024-03-18T15:59:22.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.134+0000 7faec047e640 1 -- 172.21.15.150:0/4279322873 wait complete. 2024-03-18T15:59:22.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:59:22.148 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- 172.21.15.150:0/3743625520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 msgr2=0x7fb750109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 --2- 172.21.15.150:0/3743625520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750109dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb748009ca0 tx=0x7fb74802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- 172.21.15.150:0/3743625520 shutdown_connections 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 --2- 172.21.15.150:0/3743625520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750109dd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 --2- 172.21.15.150:0/3743625520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb7500ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 --2- 172.21.15.150:0/3743625520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 0x7fb750103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- 172.21.15.150:0/3743625520 >> 172.21.15.150:0/3743625520 conn(0x7fb7500fb3e0 msgr2=0x7fb7500fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- 172.21.15.150:0/3743625520 shutdown_connections 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- 172.21.15.150:0/3743625520 wait complete. 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 Processor -- start 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 -- start start 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.251+0000 7fb7581b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb750197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 0x7fb75019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb75010d470 con 0x7fb750100220 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb75010d2f0 con 0x7fb7500ff860 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb75010d5f0 con 0x7fb750102df0 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39418/0 (socket says 172.21.15.150:39418) 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 -- 172.21.15.150:0/1944525546 learned_addr learned my addr 172.21.15.150:0/1944525546 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb755f4e640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb750197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75674f640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 0x7fb75019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 -- 172.21.15.150:0/1944525546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 msgr2=0x7fb75019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 0x7fb75019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 -- 172.21.15.150:0/1944525546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 msgr2=0x7fb750197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb750197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 -- 172.21.15.150:0/1944525546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75019f6d0 con 0x7fb750100220 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb755f4e640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb750197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb75574d640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb74000e9f0 tx=0x7fb74000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74000cd80 con 0x7fb750100220 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb75019f9c0 con 0x7fb750100220 2024-03-18T15:59:22.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb740010910 con 0x7fb750100220 2024-03-18T15:59:22.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7400188f0 con 0x7fb750100220 2024-03-18T15:59:22.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.252+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb75019ff10 con 0x7fb750100220 2024-03-18T15:59:22.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.253+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb750108c50 con 0x7fb750100220 2024-03-18T15:59:22.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.254+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb740010ac0 con 0x7fb750100220 2024-03-18T15:59:22.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.255+0000 7fb746ffd640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 0x7fb73c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.255+0000 7fb755f4e640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 0x7fb73c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.255+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb7400cef70 con 0x7fb750100220 2024-03-18T15:59:22.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.255+0000 7fb755f4e640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 0x7fb73c0a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb750069b10 tx=0x7fb74c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.259+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb74009b890 con 0x7fb750100220 2024-03-18T15:59:22.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.432+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb75002d420 con 0x7fb750100220 2024-03-18T15:59:22.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.433+0000 7fb746ffd640 1 -- 172.21.15.150:0/1944525546 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fb740084e20 con 0x7fb750100220 2024-03-18T15:59:22.435 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:59:22.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 msgr2=0x7fb73c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 0x7fb73c0a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb750069b10 tx=0x7fb74c005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:22.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 msgr2=0x7fb750197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb74000e9f0 tx=0x7fb74000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 shutdown_connections 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb73c0a2890 0x7fb73c0a4d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb750102df0 0x7fb75019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb750100220 0x7fb750197ff0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 --2- 172.21.15.150:0/1944525546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7500ff860 0x7fb750197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 >> 172.21.15.150:0/1944525546 conn(0x7fb7500fb3e0 msgr2=0x7fb750105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 shutdown_connections 2024-03-18T15:59:22.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.435+0000 7fb7581b2640 1 -- 172.21.15.150:0/1944525546 wait complete. 2024-03-18T15:59:22.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:59:22.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 -- 172.21.15.150:0/101000832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 msgr2=0x7f9108101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 --2- 172.21.15.150:0/101000832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f9108101d60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009900 tx=0x7f90f002bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:22.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 -- 172.21.15.150:0/101000832 shutdown_connections 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 --2- 172.21.15.150:0/101000832 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 0x7f9108108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 --2- 172.21.15.150:0/101000832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f9108101d60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 --2- 172.21.15.150:0/101000832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 0x7f9108100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 -- 172.21.15.150:0/101000832 >> 172.21.15.150:0/101000832 conn(0x7f91080fbc80 msgr2=0x7f91080fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 -- 172.21.15.150:0/101000832 shutdown_connections 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 -- 172.21.15.150:0/101000832 wait complete. 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.552+0000 7f910cef8640 1 Processor -- start 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 -- start start 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 0x7f910806a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 0x7f910819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f910810c370 con 0x7f91081018e0 2024-03-18T15:59:22.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f910810c1f0 con 0x7f91081006f0 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910cef8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f910810c4f0 con 0x7f91081022a0 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51124/0 (socket says 172.21.15.150:51124) 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 -- 172.21.15.150:0/2823933555 learned_addr learned my addr 172.21.15.150:0/2823933555 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f9106d9d640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 0x7f910819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f910659c640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 0x7f910806a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 -- 172.21.15.150:0/2823933555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 msgr2=0x7f910819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 0x7f910819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 -- 172.21.15.150:0/2823933555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 msgr2=0x7f910806a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 0x7f910806a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 -- 172.21.15.150:0/2823933555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910819f720 con 0x7f91081018e0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fdd9b640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009230 tx=0x7f90f002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.553+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0041070 con 0x7f91081018e0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.554+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f0004450 con 0x7f91081018e0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.554+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910819f9b0 con 0x7f91081018e0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.554+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0003e20 con 0x7f91081018e0 2024-03-18T15:59:22.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.554+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910819fea0 con 0x7f91081018e0 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.555+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9108107ab0 con 0x7f91081018e0 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.556+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f90f0044070 con 0x7f91081018e0 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.556+0000 7f90fffff640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 0x7f90f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.556+0000 7f910659c640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 0x7f90f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.556+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f90f00f09b0 con 0x7f91081018e0 2024-03-18T15:59:22.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.557+0000 7f910659c640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 0x7f90f40a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9108101740 tx=0x7f90f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.560+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90f00bd3e0 con 0x7f91081018e0 2024-03-18T15:59:22.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.735+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f910802d400 con 0x7f91081018e0 2024-03-18T15:59:22.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.736+0000 7f90fffff640 1 -- 172.21.15.150:0/2823933555 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f90f00c1480 con 0x7f91081018e0 2024-03-18T15:59:22.738 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.737+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 msgr2=0x7f90f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.737+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 0x7f90f40a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9108101740 tx=0x7f90f8005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.737+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 msgr2=0x7f910819c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.737+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009230 tx=0x7f90f002c650 comp rx=0 tx=0).stop 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 shutdown_connections 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f90f40a2890 0x7f90f40a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91081022a0 0x7f910819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91081018e0 0x7f910819c7f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 --2- 172.21.15.150:0/2823933555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91081006f0 0x7f910806a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 >> 172.21.15.150:0/2823933555 conn(0x7f91080fbc80 msgr2=0x7f9108104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 shutdown_connections 2024-03-18T15:59:22.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.738+0000 7f910cef8640 1 -- 172.21.15.150:0/2823933555 wait complete. 2024-03-18T15:59:22.750 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:22 smithi150 ceph-mon[31946]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:22.750 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:22 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4279322873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:22.750 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:22 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1944525546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:22.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:59:22.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- 172.21.15.150:0/1687448011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 msgr2=0x7fbba0101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 --2- 172.21.15.150:0/1687448011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0101d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009860 tx=0x7fbb9002bb70 comp rx=0 tx=0).stop 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- 172.21.15.150:0/1687448011 shutdown_connections 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 --2- 172.21.15.150:0/1687448011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba01022b0 0x7fbba0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 --2- 172.21.15.150:0/1687448011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0101d70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 --2- 172.21.15.150:0/1687448011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba0100700 0x7fbba0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- 172.21.15.150:0/1687448011 >> 172.21.15.150:0/1687448011 conn(0x7fbba00fbc90 msgr2=0x7fbba00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- 172.21.15.150:0/1687448011 shutdown_connections 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- 172.21.15.150:0/1687448011 wait complete. 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 Processor -- start 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.853+0000 7fbba5619640 1 -- start start 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba01022b0 0x7fbba019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbba010c380 con 0x7fbba0100700 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbba010c200 con 0x7fbba01022b0 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbba010c500 con 0x7fbba01018f0 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51142/0 (socket says 172.21.15.150:51142) 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9e7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 -- 172.21.15.150:0/3799880762 learned_addr learned my addr 172.21.15.150:0/3799880762 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9f7fe640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba01022b0 0x7fbba019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 -- 172.21.15.150:0/3799880762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 msgr2=0x7fbba0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 -- 172.21.15.150:0/3799880762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba01022b0 msgr2=0x7fbba019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba01022b0 0x7fbba019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:22.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 -- 172.21.15.150:0/3799880762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba019f750 con 0x7fbba0100700 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9e7fc640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb9effd640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400e9c0 tx=0x7fbb9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9400cd80 con 0x7fbba0100700 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb940035f0 con 0x7fbba0100700 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba019fa40 con 0x7fbba0100700 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94010690 con 0x7fbba0100700 2024-03-18T15:59:22.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.854+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba019ff10 con 0x7fbba0100700 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.855+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba0068a20 con 0x7fbba0100700 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.856+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fbb940037a0 con 0x7fbba0100700 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.857+0000 7fbb87fff640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.857+0000 7fbb9e7fc640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.857+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbb940cef20 con 0x7fbba0100700 2024-03-18T15:59:22.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.857+0000 7fbb9e7fc640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 0x7fbb880a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbb9002c180 tx=0x7fbb9002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:22.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:22.861+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb9409be90 con 0x7fbba0100700 2024-03-18T15:59:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:22 smithi191 ceph-mon[36784]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:22 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4279322873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:22.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:22 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1944525546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:22 smithi156 ceph-mon[35919]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:23.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:22 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4279322873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:23.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:22 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1944525546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.033+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbba01a0330 con 0x7fbba0100700 2024-03-18T15:59:23.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.034+0000 7fbb87fff640 1 -- 172.21.15.150:0/3799880762 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fbb9409b790 con 0x7fbba0100700 2024-03-18T15:59:23.036 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.035+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 msgr2=0x7fbb880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.035+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 0x7fbb880a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbb9002c180 tx=0x7fbb9002e040 comp rx=0 tx=0).stop 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.035+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 msgr2=0x7fbba0197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.035+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400e9c0 tx=0x7fbb9400ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 shutdown_connections 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbb880a2890 0x7fbb880a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbba01022b0 0x7fbba019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbba01018f0 0x7fbba0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 --2- 172.21.15.150:0/3799880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbba0100700 0x7fbba0197b30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 >> 172.21.15.150:0/3799880762 conn(0x7fbba00fbc90 msgr2=0x7fbba0069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 shutdown_connections 2024-03-18T15:59:23.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.036+0000 7fbba5619640 1 -- 172.21.15.150:0/3799880762 wait complete. 2024-03-18T15:59:23.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 -- 172.21.15.150:0/1897453451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c101370 msgr2=0x7f9c4c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 --2- 172.21.15.150:0/1897453451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c101370 0x7f9c4c1017f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009900 tx=0x7f9c4402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 -- 172.21.15.150:0/1897453451 shutdown_connections 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 --2- 172.21.15.150:0/1897453451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c101370 0x7f9c4c1017f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 --2- 172.21.15.150:0/1897453451 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 0x7f9c4c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 --2- 172.21.15.150:0/1897453451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c100070 0x7f9c4c100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 -- 172.21.15.150:0/1897453451 >> 172.21.15.150:0/1897453451 conn(0x7f9c4c0fbc70 msgr2=0x7f9c4c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 -- 172.21.15.150:0/1897453451 shutdown_connections 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.148+0000 7f9c51777640 1 -- 172.21.15.150:0/1897453451 wait complete. 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 Processor -- start 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 -- start start 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 0x7f9c4c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c101370 0x7f9c4c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10f670 con 0x7f9c4c100070 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10f4f0 con 0x7f9c4c101370 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c51777640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10f7f0 con 0x7f9c4c1009d0 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51168/0 (socket says 172.21.15.150:51168) 2024-03-18T15:59:23.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 -- 172.21.15.150:0/528309783 learned_addr learned my addr 172.21.15.150:0/528309783 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4a7fc640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 0x7f9c4c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4b7fe640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c101370 0x7f9c4c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 -- 172.21.15.150:0/528309783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 msgr2=0x7f9c4c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 0x7f9c4c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 -- 172.21.15.150:0/528309783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c101370 msgr2=0x7f9c4c1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c101370 0x7f9c4c1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.149+0000 7f9c4affd640 1 -- 172.21.15.150:0/528309783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c1a18d0 con 0x7f9c4c100070 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c4affd640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00ebc0 tx=0x7f9c3c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c010630 con 0x7f9c4c100070 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c3c010cb0 con 0x7f9c4c100070 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c1a1bc0 con 0x7f9c4c100070 2024-03-18T15:59:23.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c018bd0 con 0x7f9c4c100070 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.150+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c1a2110 con 0x7f9c4c100070 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.151+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4c10af10 con 0x7f9c4c100070 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.152+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9c3c0107d0 con 0x7f9c4c100070 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.152+0000 7f9c33fff640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 0x7f9c340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.152+0000 7f9c4a7fc640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 0x7f9c340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.152+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f9c3c0cf1a0 con 0x7f9c4c100070 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.153+0000 7f9c4a7fc640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 0x7f9c340a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c19b190 tx=0x7f9c40005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.157+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c3c09bac0 con 0x7f9c4c100070 2024-03-18T15:59:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.370+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c4c0623e0 con 0x7f9c4c100070 2024-03-18T15:59:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.370+0000 7f9c33fff640 1 -- 172.21.15.150:0/528309783 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f9c3c09fb60 con 0x7f9c4c100070 2024-03-18T15:59:23.373 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:23.373 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:59:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.372+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 msgr2=0x7f9c340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.372+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 0x7f9c340a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c19b190 tx=0x7f9c40005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.372+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 msgr2=0x7f9c4c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00ebc0 tx=0x7f9c3c00c6a0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 shutdown_connections 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9c340a2890 0x7f9c340a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c4c101370 0x7f9c4c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c4c1009d0 0x7f9c4c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 --2- 172.21.15.150:0/528309783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c100070 0x7f9c4c199c90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 >> 172.21.15.150:0/528309783 conn(0x7f9c4c0fbc70 msgr2=0x7f9c4c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 shutdown_connections 2024-03-18T15:59:23.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.373+0000 7f9c51777640 1 -- 172.21.15.150:0/528309783 wait complete. 2024-03-18T15:59:23.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- 172.21.15.150:0/619920253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f681018f0 msgr2=0x7f6f68101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- 172.21.15.150:0/619920253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f681018f0 0x7f6f68101d70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009900 tx=0x7f6f6402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- 172.21.15.150:0/619920253 shutdown_connections 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- 172.21.15.150:0/619920253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 0x7f6f68108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- 172.21.15.150:0/619920253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f681018f0 0x7f6f68101d70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- 172.21.15.150:0/619920253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f68100700 0x7f6f68100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- 172.21.15.150:0/619920253 >> 172.21.15.150:0/619920253 conn(0x7f6f680fbc70 msgr2=0x7f6f680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- 172.21.15.150:0/619920253 shutdown_connections 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- 172.21.15.150:0/619920253 wait complete. 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 Processor -- start 2024-03-18T15:59:23.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- start start 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f681018f0 0x7f6f6819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 0x7f6f6819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810c360 con 0x7f6f68100700 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810c1e0 con 0x7f6f681022b0 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.488+0000 7f6f70352640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810c4e0 con 0x7f6f681018f0 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51188/0 (socket says 172.21.15.150:51188) 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 -- 172.21.15.150:0/4229811742 learned_addr learned my addr 172.21.15.150:0/4229811742 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6d8ed640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f681018f0 0x7f6f6819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e8ef640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 0x7f6f6819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 -- 172.21.15.150:0/4229811742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f681018f0 msgr2=0x7f6f6819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f681018f0 0x7f6f6819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 -- 172.21.15.150:0/4229811742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 msgr2=0x7f6f6819ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 0x7f6f6819ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 -- 172.21.15.150:0/4229811742 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6819f700 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f6e0ee640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00e9c0 tx=0x7f6f5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c00cd80 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f5c0035f0 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c010690 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6819f9f0 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.489+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6819ff10 con 0x7f6f68100700 2024-03-18T15:59:23.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.490+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f68068a20 con 0x7f6f68100700 2024-03-18T15:59:23.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.491+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6f5c0037a0 con 0x7f6f68100700 2024-03-18T15:59:23.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.492+0000 7f6f5b7fe640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 0x7f6f540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.492+0000 7f6f6d8ed640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 0x7f6f540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.492+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6f5c0cefb0 con 0x7f6f68100700 2024-03-18T15:59:23.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.492+0000 7f6f6d8ed640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 0x7f6f540a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6f6402c7b0 tx=0x7f6f64005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.495+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f5c09c050 con 0x7f6f68100700 2024-03-18T15:59:23.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.672+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f68107a20 con 0x7f6f68100700 2024-03-18T15:59:23.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.673+0000 7f6f5b7fe640 1 -- 172.21.15.150:0/4229811742 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f6f5c09b950 con 0x7f6f68100700 2024-03-18T15:59:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:23.676 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:23.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 msgr2=0x7f6f540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 0x7f6f540a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6f6402c7b0 tx=0x7f6f64005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 msgr2=0x7f6f6806a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00e9c0 tx=0x7f6f5c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 shutdown_connections 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6f540a2890 0x7f6f540a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f681022b0 0x7f6f6819ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f681018f0 0x7f6f6819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 --2- 172.21.15.150:0/4229811742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f68100700 0x7f6f6806a9e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 >> 172.21.15.150:0/4229811742 conn(0x7f6f680fbc70 msgr2=0x7f6f68104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 shutdown_connections 2024-03-18T15:59:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.676+0000 7f6f70352640 1 -- 172.21.15.150:0/4229811742 wait complete. 2024-03-18T15:59:23.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:23 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2823933555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:23 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3799880762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:23 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/528309783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:23.688 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:23 smithi150 ceph-mon[31946]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:23.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:59:23.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 -- 172.21.15.150:0/205175353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1033b0 msgr2=0x7fc74c109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 --2- 172.21.15.150:0/205175353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1033b0 0x7fc74c109c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc744009ca0 tx=0x7fc74402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 -- 172.21.15.150:0/205175353 shutdown_connections 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 --2- 172.21.15.150:0/205175353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1033b0 0x7fc74c109c40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 --2- 172.21.15.150:0/205175353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1029f0 0x7fc74c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 --2- 172.21.15.150:0/205175353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 -- 172.21.15.150:0/205175353 >> 172.21.15.150:0/205175353 conn(0x7fc74c0fcd70 msgr2=0x7fc74c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 -- 172.21.15.150:0/205175353 shutdown_connections 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.792+0000 7fc75351b640 1 -- 172.21.15.150:0/205175353 wait complete. 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 Processor -- start 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 -- start start 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1033b0 0x7fc74c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c10d480 con 0x7fc74c1029f0 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c10d300 con 0x7fc74c101800 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc75351b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c10d600 con 0x7fc74c1033b0 2024-03-18T15:59:23.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51218/0 (socket says 172.21.15.150:51218) 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 -- 172.21.15.150:0/2430028438 learned_addr learned my addr 172.21.15.150:0/2430028438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc7512b7640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc751ab8640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1033b0 0x7fc74c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 -- 172.21.15.150:0/2430028438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1033b0 msgr2=0x7fc74c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1033b0 0x7fc74c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 -- 172.21.15.150:0/2430028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 msgr2=0x7fc74c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc750ab6640 1 -- 172.21.15.150:0/2430028438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0749a0 con 0x7fc74c1029f0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.793+0000 7fc7512b7640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc750ab6640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e9f0 tx=0x7fc73c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00cd80 con 0x7fc74c1029f0 2024-03-18T15:59:23.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73c005500 con 0x7fc74c1029f0 2024-03-18T15:59:23.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c1a1bb0 con 0x7fc74c1029f0 2024-03-18T15:59:23.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c010690 con 0x7fc74c1029f0 2024-03-18T15:59:23.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.794+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c1a2100 con 0x7fc74c1029f0 2024-03-18T15:59:23.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.795+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c108b20 con 0x7fc74c1029f0 2024-03-18T15:59:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.796+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc73c0046c0 con 0x7fc74c1029f0 2024-03-18T15:59:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.796+0000 7fc7427fc640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 0x7fc7380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.796+0000 7fc7512b7640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 0x7fc7380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.797+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc73c0cef70 con 0x7fc74c1029f0 2024-03-18T15:59:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.797+0000 7fc7512b7640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 0x7fc7380a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc74c102850 tx=0x7fc748005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:23.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.801+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc73c09bee0 con 0x7fc74c1029f0 2024-03-18T15:59:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:23 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2823933555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:23 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3799880762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:23 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/528309783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:23 smithi191 ceph-mon[36784]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.975+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc74c02d3d0 con 0x7fc74c1029f0 2024-03-18T15:59:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.976+0000 7fc7427fc640 1 -- 172.21.15.150:0/2430028438 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fc73c09b7e0 con 0x7fc74c1029f0 2024-03-18T15:59:23.978 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.977+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 msgr2=0x7fc7380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.977+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 0x7fc7380a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc74c102850 tx=0x7fc748005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.977+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 msgr2=0x7fc74c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.977+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e9f0 tx=0x7fc73c00edf0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 shutdown_connections 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc7380a2910 0x7fc7380a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc74c1033b0 0x7fc74c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c1029f0 0x7fc74c073bd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 --2- 172.21.15.150:0/2430028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc74c101800 0x7fc74c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 >> 172.21.15.150:0/2430028438 conn(0x7fc74c0fcd70 msgr2=0x7fc74c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 shutdown_connections 2024-03-18T15:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:23.978+0000 7fc75351b640 1 -- 172.21.15.150:0/2430028438 wait complete. 2024-03-18T15:59:23.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:59:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:23 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2823933555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:23 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3799880762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:23 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/528309783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:23 smithi156 ceph-mon[35919]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 -- 172.21.15.150:0/3285192436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c104ba0 msgr2=0x7fea0c106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3285192436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c104ba0 0x7fea0c106f90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009900 tx=0x7fe9fc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 -- 172.21.15.150:0/3285192436 shutdown_connections 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3285192436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c1074d0 0x7fea0c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3285192436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c104ba0 0x7fea0c106f90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3285192436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c102270 0x7fea0c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 -- 172.21.15.150:0/3285192436 >> 172.21.15.150:0/3285192436 conn(0x7fea0c0fbc80 msgr2=0x7fea0c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 -- 172.21.15.150:0/3285192436 shutdown_connections 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.097+0000 7fea13b3b640 1 -- 172.21.15.150:0/3285192436 wait complete. 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 Processor -- start 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 -- start start 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c104ba0 0x7fea0c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c1074d0 0x7fea0c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c10d460 con 0x7fea0c102270 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c10d2e0 con 0x7fea0c104ba0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea13b3b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fea0c10d5e0 con 0x7fea0c1074d0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51232/0 (socket says 172.21.15.150:51232) 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 -- 172.21.15.150:0/3265061754 learned_addr learned my addr 172.21.15.150:0/3265061754 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea110d6640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c104ba0 0x7fea0c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea120d8640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c1074d0 0x7fea0c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 -- 172.21.15.150:0/3265061754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c1074d0 msgr2=0x7fea0c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c1074d0 0x7fea0c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 -- 172.21.15.150:0/3265061754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c104ba0 msgr2=0x7fea0c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c104ba0 0x7fea0c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 -- 172.21.15.150:0/3265061754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0c19f760 con 0x7fea0c102270 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.098+0000 7fea118d7640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9f0 tx=0x7fea0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.099+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0800ccc0 con 0x7fea0c102270 2024-03-18T15:59:24.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.099+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea08010910 con 0x7fea0c102270 2024-03-18T15:59:24.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.099+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0c19fa50 con 0x7fea0c102270 2024-03-18T15:59:24.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.099+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea080188a0 con 0x7fea0c102270 2024-03-18T15:59:24.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.099+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea0c19ffc0 con 0x7fea0c102270 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.100+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea0c0fd770 con 0x7fea0c102270 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.101+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fea08010ac0 con 0x7fea0c102270 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.101+0000 7fea02ffd640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 0x7fe9f80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.101+0000 7fea110d6640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 0x7fe9f80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.101+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fea080cef60 con 0x7fea0c102270 2024-03-18T15:59:24.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.102+0000 7fea110d6640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 0x7fe9f80a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc000c00 tx=0x7fe9fc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.106+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea0809bed0 con 0x7fea0c102270 2024-03-18T15:59:24.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.285+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea0c0623e0 con 0x7fea0c102270 2024-03-18T15:59:24.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.285+0000 7fea02ffd640 1 -- 172.21.15.150:0/3265061754 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fea0809b7d0 con 0x7fea0c102270 2024-03-18T15:59:24.287 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 msgr2=0x7fe9f80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 0x7fe9f80a4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc000c00 tx=0x7fe9fc005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 msgr2=0x7fea0c197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9f0 tx=0x7fea0800edf0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 shutdown_connections 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe9f80a2910 0x7fe9f80a4dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea0c1074d0 0x7fea0c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea0c104ba0 0x7fea0c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 --2- 172.21.15.150:0/3265061754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea0c102270 0x7fea0c197b40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 >> 172.21.15.150:0/3265061754 conn(0x7fea0c0fbc80 msgr2=0x7fea0c102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 shutdown_connections 2024-03-18T15:59:24.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.287+0000 7fea13b3b640 1 -- 172.21.15.150:0/3265061754 wait complete. 2024-03-18T15:59:24.301 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 -- 172.21.15.150:0/1478574768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde10100700 msgr2=0x7fde10100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 --2- 172.21.15.150:0/1478574768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde10100700 0x7fde10100b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fde00009900 tx=0x7fde0002bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 -- 172.21.15.150:0/1478574768 shutdown_connections 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 --2- 172.21.15.150:0/1478574768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde101022b0 0x7fde10108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 --2- 172.21.15.150:0/1478574768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 0x7fde10101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 --2- 172.21.15.150:0/1478574768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde10100700 0x7fde10100b20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 -- 172.21.15.150:0/1478574768 >> 172.21.15.150:0/1478574768 conn(0x7fde100fbc90 msgr2=0x7fde100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 -- 172.21.15.150:0/1478574768 shutdown_connections 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 -- 172.21.15.150:0/1478574768 wait complete. 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.403+0000 7fde15cf8640 1 Processor -- start 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 -- start start 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 0x7fde10197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 0x7fde101980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010d460 con 0x7fde101022b0 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010d2e0 con 0x7fde101018f0 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde15cf8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010d5e0 con 0x7fde10100700 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51240/0 (socket says 172.21.15.150:51240) 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 -- 172.21.15.150:0/303261890 learned_addr learned my addr 172.21.15.150:0/303261890 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 0x7fde10197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0effd640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 0x7fde101980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 -- 172.21.15.150:0/303261890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 msgr2=0x7fde10197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 0x7fde10197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 -- 172.21.15.150:0/303261890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 msgr2=0x7fde101980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 0x7fde101980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 -- 172.21.15.150:0/303261890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1019f7b0 con 0x7fde101022b0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0f7fe640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 0x7fde10197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.404+0000 7fde0ffff640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde0800efd0 tx=0x7fde0800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.405+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08003b70 con 0x7fde101022b0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.405+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1019faa0 con 0x7fde101022b0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.405+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde08003d10 con 0x7fde101022b0 2024-03-18T15:59:24.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.405+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde080109f0 con 0x7fde101022b0 2024-03-18T15:59:24.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.405+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1019ffa0 con 0x7fde101022b0 2024-03-18T15:59:24.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.406+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde10107a90 con 0x7fde101022b0 2024-03-18T15:59:24.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.407+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fde08004020 con 0x7fde101022b0 2024-03-18T15:59:24.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.407+0000 7fde0cff9640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 0x7fddfc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.407+0000 7fde0f7fe640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 0x7fddfc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.407+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fde080cef90 con 0x7fde101022b0 2024-03-18T15:59:24.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.408+0000 7fde0f7fe640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 0x7fddfc0a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde00000c00 tx=0x7fde00005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.412+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde0809bfa0 con 0x7fde101022b0 2024-03-18T15:59:24.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.588+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde1002d390 con 0x7fde101022b0 2024-03-18T15:59:24.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.589+0000 7fde0cff9640 1 -- 172.21.15.150:0/303261890 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fde0809b8a0 con 0x7fde101022b0 2024-03-18T15:59:24.591 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 msgr2=0x7fddfc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 0x7fddfc0a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde00000c00 tx=0x7fde00005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 msgr2=0x7fde1019f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde0800efd0 tx=0x7fde0800c6a0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 shutdown_connections 2024-03-18T15:59:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fddfc0a2890 0x7fddfc0a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde101022b0 0x7fde1019f140 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde101018f0 0x7fde101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 --2- 172.21.15.150:0/303261890 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde10100700 0x7fde10197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 >> 172.21.15.150:0/303261890 conn(0x7fde100fbc90 msgr2=0x7fde10104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 shutdown_connections 2024-03-18T15:59:24.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.591+0000 7fde15cf8640 1 -- 172.21.15.150:0/303261890 wait complete. 2024-03-18T15:59:24.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:59:24.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 -- 172.21.15.150:0/48382129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 msgr2=0x7f2750109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 --2- 172.21.15.150:0/48382129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f2750109c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2748009ca0 tx=0x7f274802c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 -- 172.21.15.150:0/48382129 shutdown_connections 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 --2- 172.21.15.150:0/48382129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f2750109c40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 --2- 172.21.15.150:0/48382129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 0x7f2750102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 --2- 172.21.15.150:0/48382129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 0x7f2750101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 -- 172.21.15.150:0/48382129 >> 172.21.15.150:0/48382129 conn(0x7f27500fcd90 msgr2=0x7f27500ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 -- 172.21.15.150:0/48382129 shutdown_connections 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.707+0000 7f2754a9a640 1 -- 172.21.15.150:0/48382129 wait complete. 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 Processor -- start 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 -- start start 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 0x7f2750197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 0x7f2750198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f275010d460 con 0x7f27501033b0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f275010d2e0 con 0x7f2750101800 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f2754a9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f275010d5e0 con 0x7f27501029f0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51258/0 (socket says 172.21.15.150:51258) 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 -- 172.21.15.150:0/3469202804 learned_addr learned my addr 172.21.15.150:0/3469202804 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274e59c640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 0x7f2750197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274dd9b640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 0x7f2750198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 -- 172.21.15.150:0/3469202804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 msgr2=0x7f2750198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 0x7f2750198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 -- 172.21.15.150:0/3469202804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 msgr2=0x7f2750197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 0x7f2750197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.708+0000 7f274ed9d640 1 -- 172.21.15.150:0/3469202804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275019f6f0 con 0x7f27501033b0 2024-03-18T15:59:24.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f274ed9d640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2748037040 tx=0x7f2748032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748041020 con 0x7f27501033b0 2024-03-18T15:59:24.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275019f980 con 0x7f27501033b0 2024-03-18T15:59:24.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2748034cd0 con 0x7f27501033b0 2024-03-18T15:59:24.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748039040 con 0x7f27501033b0 2024-03-18T15:59:24.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.709+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275019fec0 con 0x7f27501033b0 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.710+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2750108b20 con 0x7f27501033b0 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.711+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2748044020 con 0x7f27501033b0 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.711+0000 7f273f7fe640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 0x7f27380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.711+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f27480f0950 con 0x7f27501033b0 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.711+0000 7f274e59c640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 0x7f27380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.712+0000 7f274e59c640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 0x7f27380a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2750102850 tx=0x7f2740005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:24.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.716+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f274804b020 con 0x7f27501033b0 2024-03-18T15:59:24.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.926+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f275002d390 con 0x7f27501033b0 2024-03-18T15:59:24.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.927+0000 7f273f7fe640 1 -- 172.21.15.150:0/3469202804 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f27480bda60 con 0x7f27501033b0 2024-03-18T15:59:24.929 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:24.929 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:59:24.930 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:24.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 msgr2=0x7f27380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 0x7f27380a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2750102850 tx=0x7f2740005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:24.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 msgr2=0x7f275019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2748037040 tx=0x7f2748032f40 comp rx=0 tx=0).stop 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 shutdown_connections 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f27380a2890 0x7f27380a4d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27501033b0 0x7f275019f080 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27501029f0 0x7f2750198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 --2- 172.21.15.150:0/3469202804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2750101800 0x7f2750197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 >> 172.21.15.150:0/3469202804 conn(0x7f27500fcd90 msgr2=0x7f2750105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 shutdown_connections 2024-03-18T15:59:24.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:24.929+0000 7f2754a9a640 1 -- 172.21.15.150:0/3469202804 wait complete. 2024-03-18T15:59:24.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:24 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4229811742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:24.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:24 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2430028438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.941 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:24 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3265061754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.942 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:24 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/303261890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:24 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4229811742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:24 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2430028438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:24 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3265061754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:24.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:24 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/303261890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:25.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:24 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4229811742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:24 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2430028438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:25.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:24 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3265061754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:25.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:24 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/303261890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- 172.21.15.150:0/3891311211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 msgr2=0x7f3f881046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/3891311211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f881046b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009900 tx=0x7f3f7c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- 172.21.15.150:0/3891311211 shutdown_connections 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/3891311211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88107520 0x7f3f88109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/3891311211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88104bf0 0x7f3f88106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/3891311211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f881046b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- 172.21.15.150:0/3891311211 >> 172.21.15.150:0/3891311211 conn(0x7f3f880fbc70 msgr2=0x7f3f880fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- 172.21.15.150:0/3891311211 shutdown_connections 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- 172.21.15.150:0/3891311211 wait complete. 2024-03-18T15:59:25.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 Processor -- start 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.042+0000 7f3f8dd76640 1 -- start start 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f88197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88107520 0x7f3f8819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8810d4a0 con 0x7f3f88104bf0 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8810d320 con 0x7f3f88107520 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8810d620 con 0x7f3f881022c0 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f877fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f88197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51274/0 (socket says 172.21.15.150:51274) 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 -- 172.21.15.150:0/2007968304 learned_addr learned my addr 172.21.15.150:0/2007968304 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f87fff640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88107520 0x7f3f8819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 -- 172.21.15.150:0/2007968304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 msgr2=0x7f3f88197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f88197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 -- 172.21.15.150:0/2007968304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88107520 msgr2=0x7f3f8819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88107520 0x7f3f8819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 -- 172.21.15.150:0/2007968304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8819f760 con 0x7f3f88104bf0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f86ffd640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e9f0 tx=0x7f3f7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800ccc0 con 0x7f3f88104bf0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8819fa50 con 0x7f3f88104bf0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f78005500 con 0x7f3f88104bf0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78010690 con 0x7f3f88104bf0 2024-03-18T15:59:25.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.043+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8819ff50 con 0x7f3f88104bf0 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.044+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f880fd760 con 0x7f3f88104bf0 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.045+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f3f780046c0 con 0x7f3f88104bf0 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.046+0000 7f3f84ff9640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 0x7f3f700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.046+0000 7f3f877fe640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 0x7f3f700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.046+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f3f780cf100 con 0x7f3f88104bf0 2024-03-18T15:59:25.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.046+0000 7f3f877fe640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 0x7f3f700a4d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c02c7b0 tx=0x7f3f7c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.050+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f78053740 con 0x7f3f88104bf0 2024-03-18T15:59:25.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.230+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f8802d1b0 con 0x7f3f88104bf0 2024-03-18T15:59:25.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.231+0000 7f3f84ff9640 1 -- 172.21.15.150:0/2007968304 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f3f8802d1b0 con 0x7f3f88104bf0 2024-03-18T15:59:25.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:25.235 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:25.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 msgr2=0x7f3f700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 0x7f3f700a4d50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c02c7b0 tx=0x7f3f7c005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 msgr2=0x7f3f88198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e9f0 tx=0x7f3f7800eeb0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 shutdown_connections 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f3f700a2890 0x7f3f700a4d50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f88107520 0x7f3f8819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.233+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f88104bf0 0x7f3f88198080 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.234+0000 7f3f8dd76640 1 --2- 172.21.15.150:0/2007968304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f881022c0 0x7f3f88197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.234+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 >> 172.21.15.150:0/2007968304 conn(0x7f3f880fbc70 msgr2=0x7f3f88102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.234+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 shutdown_connections 2024-03-18T15:59:25.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.234+0000 7f3f8dd76640 1 -- 172.21.15.150:0/2007968304 wait complete. 2024-03-18T15:59:25.248 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:59:25.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 -- 172.21.15.150:0/2531834608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 msgr2=0x7f5834101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 --2- 172.21.15.150:0/2531834608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834101d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5824009900 tx=0x7f582402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 -- 172.21.15.150:0/2531834608 shutdown_connections 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 --2- 172.21.15.150:0/2531834608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58341022c0 0x7f5834108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 --2- 172.21.15.150:0/2531834608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834101d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 --2- 172.21.15.150:0/2531834608 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5834100710 0x7f5834100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 -- 172.21.15.150:0/2531834608 >> 172.21.15.150:0/2531834608 conn(0x7f58340fbcc0 msgr2=0x7f58340fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 -- 172.21.15.150:0/2531834608 shutdown_connections 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.348+0000 7f583c627640 1 -- 172.21.15.150:0/2531834608 wait complete. 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 Processor -- start 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 -- start start 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5834100710 0x7f5834195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58341022c0 0x7f583419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f583410b2e0 con 0x7f5834101900 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f583410b160 con 0x7f5834100710 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583c627640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f583410b460 con 0x7f58341022c0 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51296/0 (socket says 172.21.15.150:51296) 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 -- 172.21.15.150:0/309025346 learned_addr learned my addr 172.21.15.150:0/309025346 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583a3c3640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5834100710 0x7f5834195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f583abc4640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58341022c0 0x7f583419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 -- 172.21.15.150:0/309025346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58341022c0 msgr2=0x7f583419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58341022c0 0x7f583419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 -- 172.21.15.150:0/309025346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5834100710 msgr2=0x7f5834195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5834100710 0x7f5834195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 -- 172.21.15.150:0/309025346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583419d580 con 0x7f5834101900 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.349+0000 7f5839bc2640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5824000c00 tx=0x7f582402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.350+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5824043020 con 0x7f5834101900 2024-03-18T15:59:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.350+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58240054e0 con 0x7f5834101900 2024-03-18T15:59:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.350+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583419d810 con 0x7f5834101900 2024-03-18T15:59:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.350+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5824003bd0 con 0x7f5834101900 2024-03-18T15:59:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.350+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583419ddb0 con 0x7f5834101900 2024-03-18T15:59:25.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.351+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5834107a30 con 0x7f5834101900 2024-03-18T15:59:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.352+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f5824046020 con 0x7f5834101900 2024-03-18T15:59:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.352+0000 7f582b7fe640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 0x7f58200a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.352+0000 7f583a3c3640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 0x7f58200a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.352+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f58240f0a80 con 0x7f5834101900 2024-03-18T15:59:25.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.353+0000 7f583a3c3640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 0x7f58200a6fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5834101760 tx=0x7f5830005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.357+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f582404d020 con 0x7f5834101900 2024-03-18T15:59:25.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.600+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583402d1b0 con 0x7f5834101900 2024-03-18T15:59:25.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.600+0000 7f582b7fe640 1 -- 172.21.15.150:0/309025346 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f58240bdb00 con 0x7f5834101900 2024-03-18T15:59:25.603 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:25.603 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:59:25.603 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:25.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 msgr2=0x7f58200a6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 0x7f58200a6fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5834101760 tx=0x7f5830005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:25.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 msgr2=0x7f5834195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5824000c00 tx=0x7f582402c650 comp rx=0 tx=0).stop 2024-03-18T15:59:25.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 shutdown_connections 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f58200a4b20 0x7f58200a6fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58341022c0 0x7f583419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5834101900 0x7f5834195ea0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 --2- 172.21.15.150:0/309025346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5834100710 0x7f5834195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 >> 172.21.15.150:0/309025346 conn(0x7f58340fbcc0 msgr2=0x7f5834104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 shutdown_connections 2024-03-18T15:59:25.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.603+0000 7f583c627640 1 -- 172.21.15.150:0/309025346 wait complete. 2024-03-18T15:59:25.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T15:57:56.512066+0000', 'modified': '2024-03-18T15:58:01.194272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24496, 'mds_1': 24485, 'mds_2': 24481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/2692248421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 2692248421}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 2692248421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/1894652569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 1894652569}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 1894652569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/2169559605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 2169559605}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 2169559605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.191:6837/720633938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 720633938}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 720633938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.156:6839/219054917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 219054917}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 219054917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/3948564183', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 3948564183}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 3948564183}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24496, 'qdb_cluster': [24481, 24485, 24496]} 2024-03-18T15:59:25.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T15:59:25.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/3694190273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec102df0 msgr2=0x7f9eec103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/3694190273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec102df0 0x7f9eec103210 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009900 tx=0x7f9edc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/3694190273 shutdown_connections 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/3694190273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec100220 0x7f9eec109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/3694190273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec0ff860 0x7f9eec0ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/3694190273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec102df0 0x7f9eec103210 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/3694190273 >> 172.21.15.150:0/3694190273 conn(0x7f9eec0fb3e0 msgr2=0x7f9eec0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/3694190273 shutdown_connections 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/3694190273 wait complete. 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 Processor -- start 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 -- start start 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.718+0000 7f9ef3eb5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec100220 0x7f9eec198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec102df0 0x7f9eec19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec10d470 con 0x7f9eec0ff860 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec10d2f0 con 0x7f9eec100220 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec10d5f0 con 0x7f9eec102df0 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51312/0 (socket says 172.21.15.150:51312) 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1935845381 learned_addr learned my addr 172.21.15.150:0/1935845381 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec102df0 0x7f9eec19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1450640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec100220 0x7f9eec198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1935845381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec102df0 msgr2=0x7f9eec19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec102df0 0x7f9eec19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1935845381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec100220 msgr2=0x7f9eec198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec100220 0x7f9eec198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1935845381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eec19f720 con 0x7f9eec0ff860 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9edc02efd0 tx=0x7f9edc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ee2ffd640 1 -- 172.21.15.150:0/1935845381 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc041070 con 0x7f9eec0ff860 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ee2ffd640 1 -- 172.21.15.150:0/1935845381 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9edc004450 con 0x7f9eec0ff860 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eec19f9b0 con 0x7f9eec0ff860 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ee2ffd640 1 -- 172.21.15.150:0/1935845381 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc034440 con 0x7f9eec0ff860 2024-03-18T15:59:25.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.719+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eec19fef0 con 0x7f9eec0ff860 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ee2ffd640 1 -- 172.21.15.150:0/1935845381 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9edc044070 con 0x7f9eec0ff860 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 Processor -- start 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 -- start start 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 0x55b2452880c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b24528abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b2451e6c80 con 0x55b245282bd0 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b24518faa0 con 0x55b245288730 2024-03-18T15:59:25.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.722+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b2451e2d20 con 0x55b2452835c0 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51314/0 (socket says 172.21.15.150:51314) 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 -- 172.21.15.150:0/519774539 learned_addr learned my addr 172.21.15.150:0/519774539 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 0x55b2452880c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef1450640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b24528abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ee2ffd640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 0x7f9ed80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 -- 172.21.15.150:0/519774539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 msgr2=0x55b2452880c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 0x55b2452880c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 -- 172.21.15.150:0/519774539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 msgr2=0x55b24528abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b24528abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 -- 172.21.15.150:0/519774539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2451b1ff0 con 0x55b245282bd0 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 0x55b2452880c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef1450640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b24528abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ee2ffd640 1 -- 172.21.15.150:0/1935845381 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f9edc0f0610 con 0x7f9eec0ff860 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 0x7f9ed80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ef2452640 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4003b10 tx=0x7f9ee4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8390a1718a6e11ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ed7fff640 1 -- 172.21.15.150:0/519774539 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4024070 con 0x55b245282bd0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ed7fff640 1 -- 172.21.15.150:0/519774539 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4010ce0 con 0x55b245282bd0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.723+0000 7f9ed7fff640 1 -- 172.21.15.150:0/519774539 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee401ed80 con 0x55b245282bd0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- 172.21.15.150:0/519774539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 msgr2=0x55b245282ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 0x7f9ed80a4dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4021000 tx=0x7f9ee401e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4003b10 tx=0x7f9ee4009f90 comp rx=0 tx=0).stop 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- 172.21.15.150:0/519774539 shutdown_connections 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b24528abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b2452835c0 0x55b2452880c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/519774539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b245282ff0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- 172.21.15.150:0/519774539 >> 172.21.15.150:0/519774539 conn(0x55b24527e140 msgr2=0x55b24527e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- 172.21.15.150:0/519774539 shutdown_connections 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- 172.21.15.150:0/519774539 wait complete. 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.724+0000 7f9ef6b76740 1 -- start start 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 Processor -- start 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b24528d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245288730 0x55b245294330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b24528b6e0 con 0x55b245282bd0 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b2451a2ef0 con 0x55b2452835c0 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b24528b860 con 0x55b245288730 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef6b76740 10 client.0 _ll_register_callbacks cb 0x55b24519ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51316/0 (socket says 172.21.15.150:51316) 2024-03-18T15:59:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1022713178 learned_addr learned my addr 172.21.15.150:0/1022713178 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1450640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b24528d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245288730 0x55b245294330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245288730 msgr2=0x55b245294330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245288730 0x55b245294330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1022713178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 msgr2=0x55b24528d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b24528d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.725+0000 7f9ef1c51640 1 -- 172.21.15.150:0/1022713178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2451afa30 con 0x55b245282bd0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0096c0 tx=0x7f9edc003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e3303b6d5dcb7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc04b070 con 0x55b245282bd0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f9edc0f2be0 con 0x55b245282bd0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee4010ce0 con 0x55b245282bd0 2024-03-18T15:59:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9edc0bcb80 con 0x55b245282bd0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b244efb2d0 con 0x55b245282bd0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4010ce0 con 0x55b245282bd0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9edc0c52a0 con 0x55b245282bd0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 10 client.15084 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 20 client.15084 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245299810 0x55b24529ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 4 client.15084 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 --> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2451f41c0 con 0x55b245299810 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.726+0000 7f9ef1450640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245299810 0x55b24529ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ed57fa640 10 client.15084 ms_handle_connect on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ef1450640 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245299810 0x55b24529ac80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16798a93310be1d8 server_cookie=2c93dcd00cbcf526 in_seq=0 out_seq=0 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ed57fa640 1 -- 172.21.15.150:0/1022713178 <== mds.2 v2:172.21.15.191:6836/720633938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2451f41c0 con 0x55b245299810 2024-03-18T15:59:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ed57fa640 10 client.15084 handle_command_reply: tid=0 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ef6b76740 1 client.15084 shutdown 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 msgr2=0x55b24528cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.727+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0096c0 tx=0x7f9edc003a20 comp rx=0 tx=0).stop 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 shutdown_connections 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245299810 0x55b24529ac80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245288730 0x55b245294330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b24528d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ed57fa640 0 client.15084 ms_handle_reset on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1022713178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245282bd0 0x55b24528cdb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 >> 172.21.15.150:0/1022713178 conn(0x55b24527e140 msgr2=0x55b24527d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 shutdown_connections 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1022713178 wait complete. 2024-03-18T15:59:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.728+0000 7f9ef6b76740 20 client.15084 trim_cache size 0 max 16384 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.745+0000 7f9ef6b76740 1 Processor -- start 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.745+0000 7f9ef6b76740 1 -- start start 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b245287da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b2452942c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b245298fc0 con 0x55b245288730 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b245299690 con 0x55b2452835c0 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b245299d60 con 0x55b245282bd0 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51320/0 (socket says 172.21.15.150:51320) 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 -- 172.21.15.150:0/1760385009 learned_addr learned my addr 172.21.15.150:0/1760385009 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b245287da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef1450640 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b2452942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 -- 172.21.15.150:0/1760385009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 msgr2=0x55b245287da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b245287da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 -- 172.21.15.150:0/1760385009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 msgr2=0x55b2452942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b2452942c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 -- 172.21.15.150:0/1760385009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b244fc56b0 con 0x55b245288730 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ef2452640 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009970 tx=0x7f9ee40096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53fecf59abee692 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ed77fe640 1 -- 172.21.15.150:0/1760385009 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4024040 con 0x55b245288730 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ed77fe640 1 -- 172.21.15.150:0/1760385009 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee40049b0 con 0x55b245288730 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.746+0000 7f9ed77fe640 1 -- 172.21.15.150:0/1760385009 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4010860 con 0x55b245288730 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1760385009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 msgr2=0x55b24527e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009970 tx=0x7f9ee40096a0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1760385009 shutdown_connections 2024-03-18T15:59:25.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b245288730 0x55b24527e520 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b2452835c0 0x55b2452942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/1760385009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b245287da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1760385009 >> 172.21.15.150:0/1760385009 conn(0x55b24527e140 msgr2=0x55b2452813a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1760385009 shutdown_connections 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- 172.21.15.150:0/1760385009 wait complete. 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 -- start start 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.747+0000 7f9ef6b76740 1 Processor -- start 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b2452901c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b245293720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55b24529ab00 con 0x55b2452835c0 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b24529a430 con 0x55b245288730 2024-03-18T15:59:25.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55b24529b1d0 con 0x55b245282bd0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 10 client.0 _ll_register_callbacks cb 0x7f9edc003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51332/0 (socket says 172.21.15.150:51332) 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1c51640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b2452901c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 -- 172.21.15.150:0/2388653729 learned_addr learned my addr 172.21.15.150:0/2388653729 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef2452640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b245293720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 -- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 msgr2=0x55b2452901c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b2452901c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 -- 172.21.15.150:0/2388653729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 msgr2=0x55b245293720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b245293720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 -- 172.21.15.150:0/2388653729 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b245293d10 con 0x55b2452835c0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef1450640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009730 tx=0x7f9ee8020560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4400ccde9112602b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8024030 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f9ee8005020 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee40049b0 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9ee8005220 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 10 client.15090 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b2452ffde0 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.748+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee40049b0 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f9ee8003710 con 0x55b2452835c0 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 10 client.15090 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 10 client.15090 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 20 client.15090 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245304a80 0x55b245303790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 4 client.15090 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 --> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] -- command(tid 0: {"prefix": "status"}) -- 0x55b2451ee510 con 0x55b245304a80 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245304a80 0x55b245303790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ed57fa640 10 client.15090 ms_handle_connect on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef1c51640 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245304a80 0x55b245303790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab1e5f9581b3de6e server_cookie=bcc8765826844deb in_seq=0 out_seq=0 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ed57fa640 1 -- 172.21.15.150:0/2388653729 <== mds.2 v2:172.21.15.191:6836/720633938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b2451ee510 con 0x55b245304a80 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ed57fa640 10 client.15090 handle_command_reply: tid=0 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 1 client.15090 shutdown 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 msgr2=0x55b245290700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.749+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009730 tx=0x7f9ee8020560 comp rx=0 tx=0).stop 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 shutdown_connections 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x55b245304a80 0x55b245303790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b245288730 0x55b245293720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2452835c0 0x55b245290700 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ed57fa640 0 client.15090 ms_handle_reset on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:25.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 --2- 172.21.15.150:0/2388653729 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55b245282bd0 0x55b2452901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 >> 172.21.15.150:0/2388653729 conn(0x55b24527e140 msgr2=0x55b245281c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 shutdown_connections 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 1 -- 172.21.15.150:0/2388653729 wait complete. 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef6b76740 20 client.15090 trim_cache size 0 max 16384 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 2, 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24481, 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 85.561541266999996, 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 109.11557836199999 2024-03-18T15:59:25.760 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 msgr2=0x7f9ed80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 0x7f9ed80a4dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4021000 tx=0x7f9ee401e000 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 msgr2=0x7f9eec197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9edc02efd0 tx=0x7f9edc02c6c0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 shutdown_connections 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f9ed80a2910 0x7f9ed80a4dd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9eec102df0 0x7f9eec19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9eec100220 0x7f9eec198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 --2- 172.21.15.150:0/1935845381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0ff860 0x7f9eec197b00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 >> 172.21.15.150:0/1935845381 conn(0x7f9eec0fb3e0 msgr2=0x7f9eec105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 shutdown_connections 2024-03-18T15:59:25.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.750+0000 7f9ef3eb5640 1 -- 172.21.15.150:0/1935845381 wait complete. 2024-03-18T15:59:25.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T15:59:25.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 -- 172.21.15.150:0/3544115183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8481033c0 msgr2=0x7fb848109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 --2- 172.21.15.150:0/3544115183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8481033c0 0x7fb848109c50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb840009ca0 tx=0x7fb84002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 -- 172.21.15.150:0/3544115183 shutdown_connections 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 --2- 172.21.15.150:0/3544115183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8481033c0 0x7fb848109c50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 --2- 172.21.15.150:0/3544115183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb848102a00 0x7fb848102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 --2- 172.21.15.150:0/3544115183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848101810 0x7fb848101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 -- 172.21.15.150:0/3544115183 >> 172.21.15.150:0/3544115183 conn(0x7fb8480fcd80 msgr2=0x7fb8480ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 -- 172.21.15.150:0/3544115183 shutdown_connections 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.864+0000 7fb84cd77640 1 -- 172.21.15.150:0/3544115183 wait complete. 2024-03-18T15:59:25.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 Processor -- start 2024-03-18T15:59:25.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 -- start start 2024-03-18T15:59:25.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848102a00 0x7fb848073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 0x7fb848074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb84810d470 con 0x7fb848101810 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb84810d2f0 con 0x7fb848102a00 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84cd77640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb84810d5f0 con 0x7fb8481033c0 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51354/0 (socket says 172.21.15.150:51354) 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 -- 172.21.15.150:0/983089872 learned_addr learned my addr 172.21.15.150:0/983089872 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb845d9b640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848102a00 0x7fb848073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb846d9d640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 0x7fb848074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 -- 172.21.15.150:0/983089872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 msgr2=0x7fb848074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 0x7fb848074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 -- 172.21.15.150:0/983089872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848102a00 msgr2=0x7fb848073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848102a00 0x7fb848073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 -- 172.21.15.150:0/983089872 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8480749a0 con 0x7fb848101810 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb846d9d640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 0x7fb848074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb84659c640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb83800e9c0 tx=0x7fb83800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb8377fe640 1 -- 172.21.15.150:0/983089872 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83800cd80 con 0x7fb848101810 2024-03-18T15:59:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.865+0000 7fb8377fe640 1 -- 172.21.15.150:0/983089872 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8380035f0 con 0x7fb848101810 2024-03-18T15:59:25.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.866+0000 7fb8377fe640 1 -- 172.21.15.150:0/983089872 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb838010690 con 0x7fb848101810 2024-03-18T15:59:25.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.866+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8481a1ba0 con 0x7fb848101810 2024-03-18T15:59:25.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.866+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8481a2140 con 0x7fb848101810 2024-03-18T15:59:25.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb8377fe640 1 -- 172.21.15.150:0/983089872 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb8380037a0 con 0x7fb848101810 2024-03-18T15:59:25.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 Processor -- start 2024-03-18T15:59:25.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 -- start start 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb8377fe640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 0x7fb8300a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 0x5629fa6aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5629fa608c80 con 0x5629fa6a4bd0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629fa5b1aa0 con 0x5629fa6aa730 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.868+0000 7fb84fa39740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5629fa604d20 con 0x5629fa6a55c0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb84659c640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 0x7fb8300a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb846d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb846d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51364/0 (socket says 172.21.15.150:51364) 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb846d9d640 1 -- 172.21.15.150:0/780784577 learned_addr learned my addr 172.21.15.150:0/780784577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 0x5629fa6aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb846d9d640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb8377fe640 1 -- 172.21.15.150:0/983089872 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb8380cf000 con 0x7fb848101810 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 -- 172.21.15.150:0/780784577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 msgr2=0x5629fa6acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 -- 172.21.15.150:0/780784577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 msgr2=0x5629fa6a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 -- 172.21.15.150:0/780784577 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629fa5d3ff0 con 0x5629fa6a55c0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb846d9d640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb84659c640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 0x7fb8300a4dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb838098240 tx=0x7fb838092040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb845d9b640 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 0x5629fa6aa0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009bf0 tx=0x7fb83c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4a08c4c3e5d644 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb834ff9640 1 -- 172.21.15.150:0/780784577 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c01e070 con 0x5629fa6a55c0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb834ff9640 1 -- 172.21.15.150:0/780784577 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c009da0 con 0x5629fa6a55c0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.869+0000 7fb834ff9640 1 -- 172.21.15.150:0/780784577 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0187a0 con 0x5629fa6a55c0 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- 172.21.15.150:0/780784577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 msgr2=0x5629fa6aa0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 0x5629fa6aa0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009bf0 tx=0x7fb83c011810 comp rx=0 tx=0).stop 2024-03-18T15:59:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- 172.21.15.150:0/780784577 shutdown_connections 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a55c0 0x5629fa6aa0c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 --2- 172.21.15.150:0/780784577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a4bd0 0x5629fa6a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- 172.21.15.150:0/780784577 >> 172.21.15.150:0/780784577 conn(0x5629fa6a0140 msgr2=0x5629fa6a0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- 172.21.15.150:0/780784577 shutdown_connections 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- 172.21.15.150:0/780784577 wait complete. 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 -- start start 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.870+0000 7fb84fa39740 1 Processor -- start 2024-03-18T15:59:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a4bd0 0x5629fa6aee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6aa730 0x5629fa6b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6ad6e0 con 0x5629fa6a55c0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629fa5c4ef0 con 0x5629fa6a4bd0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6ad860 con 0x5629fa6aa730 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 10 client.0 _ll_register_callbacks cb 0x5629fa5bfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb846d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a4bd0 0x5629fa6aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51368/0 (socket says 172.21.15.150:51368) 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 -- 172.21.15.150:0/99060293 learned_addr learned my addr 172.21.15.150:0/99060293 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb846d9d640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6aa730 0x5629fa6b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 -- 172.21.15.150:0/99060293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6aa730 msgr2=0x5629fa6b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6aa730 0x5629fa6b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 -- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a4bd0 msgr2=0x5629fa6aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a4bd0 0x5629fa6aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 -- 172.21.15.150:0/99060293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629fa5d1a30 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb845d9b640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb83c01c040 tx=0x7fb83c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346fdf63e7a777fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c01e040 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb83c005040 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.871+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb83c009da0 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb83c005240 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 10 client.15102 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5629fa31d2d0 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c009da0 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb83c021030 con 0x5629fa6a55c0 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 10 client.15102 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 10 client.15102 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:59:25.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 20 client.15102 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa6bb560 0x5629fa6bc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 4 client.15102 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 --> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629fa6161c0 con 0x5629fa6bb560 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb846d9d640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa6bb560 0x5629fa6bc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb82dffb640 10 client.15102 ms_handle_connect on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.872+0000 7fb846d9d640 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa6bb560 0x5629fa6bc9d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9dbfc55ebb73ea0 server_cookie=e4b32794490de981 in_seq=0 out_seq=0 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb82dffb640 1 -- 172.21.15.150:0/99060293 <== mds.1 v2:172.21.15.156:6838/219054917 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629fa6161c0 con 0x5629fa6bb560 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb82dffb640 10 client.15102 handle_command_reply: tid=0 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 client.15102 shutdown 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 msgr2=0x5629fa6af350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb83c01c040 tx=0x7fb83c009850 comp rx=0 tx=0).stop 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 shutdown_connections 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa6bb560 0x5629fa6bc9d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6aa730 0x5629fa6b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6af350 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 --2- 172.21.15.150:0/99060293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a4bd0 0x5629fa6aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb82dffb640 0 client.15102 ms_handle_reset on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 >> 172.21.15.150:0/99060293 conn(0x5629fa6a0140 msgr2=0x5629fa69f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 shutdown_connections 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 1 -- 172.21.15.150:0/99060293 wait complete. 2024-03-18T15:59:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.873+0000 7fb84fa39740 20 client.15102 trim_cache size 0 max 16384 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 Processor -- start 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 -- start start 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bad10 con 0x5629fa6a55c0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bb3e0 con 0x5629fa6aa730 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb84fa39740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bbab0 con 0x5629fa6a4bd0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb845d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb845d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51382/0 (socket says 172.21.15.150:51382) 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb845d9b640 1 -- 172.21.15.150:0/1784500474 learned_addr learned my addr 172.21.15.150:0/1784500474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.891+0000 7fb846d9d640 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb846d9d640 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 -- 172.21.15.150:0/1784500474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 msgr2=0x5629fa6adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 -- 172.21.15.150:0/1784500474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 msgr2=0x5629fa6a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 -- 172.21.15.150:0/1784500474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629fa3e76b0 con 0x5629fa6a55c0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb845d9b640 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb83c011850 tx=0x7fb83c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bd0a4558ece96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb82ffff640 1 -- 172.21.15.150:0/1784500474 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c01e040 con 0x5629fa6a55c0 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb82ffff640 1 -- 172.21.15.150:0/1784500474 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c029070 con 0x5629fa6a55c0 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.892+0000 7fb82ffff640 1 -- 172.21.15.150:0/1784500474 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c003ad0 con 0x5629fa6a55c0 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- 172.21.15.150:0/1784500474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 msgr2=0x5629fa6b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb83c011850 tx=0x7fb83c011d10 comp rx=0 tx=0).stop 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- 172.21.15.150:0/1784500474 shutdown_connections 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6aa730 0x5629fa6a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6a55c0 0x5629fa6b6350 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- 172.21.15.150:0/1784500474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- 172.21.15.150:0/1784500474 >> 172.21.15.150:0/1784500474 conn(0x5629fa6a0140 msgr2=0x5629fa6ba530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- 172.21.15.150:0/1784500474 shutdown_connections 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- 172.21.15.150:0/1784500474 wait complete. 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- start start 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 Processor -- start 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a55c0 0x5629fa6b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bc850 con 0x5629fa6aa730 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bc180 con 0x5629fa6a55c0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5629fa6bcf20 con 0x5629fa6a4bd0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb84fa39740 10 client.0 _ll_register_callbacks cb 0x7fb83c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb846d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb846d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51384/0 (socket says 172.21.15.150:51384) 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb846d9d640 1 -- 172.21.15.150:0/124352826 learned_addr learned my addr 172.21.15.150:0/124352826 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb845d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a55c0 0x5629fa6b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.893+0000 7fb846d9d640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 -- 172.21.15.150:0/124352826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 msgr2=0x5629fa6b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 -- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a55c0 msgr2=0x5629fa6b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:25.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a55c0 0x5629fa6b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 -- 172.21.15.150:0/124352826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629fa6b51a0 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb846d9d640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb84002c860 tx=0x7fb83c004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f886f325e5728c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840039040 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb84004dc00 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb83c029070 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb84004de00 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 10 client.15108 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5629fa6b6510 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83c029070 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb840005020 con 0x5629fa6aa730 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 10 client.15108 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 10 client.15108 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:59:25.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 20 client.15108 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa725b60 0x5629fa721fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 4 client.15108 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.894+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 --> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] -- command(tid 0: {"prefix": "status"}) -- 0x5629fa610510 con 0x5629fa725b60 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb845d9b640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa725b60 0x5629fa721fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb845d9b640 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa725b60 0x5629fa721fa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fba96cc971a86306 server_cookie=888257eea8aece02 in_seq=0 out_seq=0 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb82dffb640 10 client.15108 ms_handle_connect on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb82dffb640 1 -- 172.21.15.150:0/124352826 <== mds.1 v2:172.21.15.156:6838/219054917 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5629fa610510 con 0x5629fa725b60 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb82dffb640 10 client.15108 handle_command_reply: tid=0 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 client.15108 shutdown 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 msgr2=0x5629fa6b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb84002c860 tx=0x7fb83c004350 comp rx=0 tx=0).stop 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 shutdown_connections 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x5629fa725b60 0x5629fa721fa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5629fa6aa730 0x5629fa6b4bb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb82dffb640 0 client.15108 ms_handle_reset on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629fa6a55c0 0x5629fa6b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 --2- 172.21.15.150:0/124352826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5629fa6a4bd0 0x5629fa6b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 >> 172.21.15.150:0/124352826 conn(0x5629fa6a0140 msgr2=0x5629fa6aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 shutdown_connections 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 1 -- 172.21.15.150:0/124352826 wait complete. 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.895+0000 7fb84fa39740 20 client.15108 trim_cache size 0 max 16384 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 1, 2024-03-18T15:59:25.904 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24485, 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 86.717607978999993, 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 94.746254477999997 2024-03-18T15:59:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 msgr2=0x7fb8300a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 0x7fb8300a4dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb838098240 tx=0x7fb838092040 comp rx=0 tx=0).stop 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 msgr2=0x7fb848075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb83800e9c0 tx=0x7fb83800ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 shutdown_connections 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8300a2910 0x7fb8300a4dd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8481033c0 0x7fb848074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb848102a00 0x7fb848073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 --2- 172.21.15.150:0/983089872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb848101810 0x7fb848075590 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:25.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 >> 172.21.15.150:0/983089872 conn(0x7fb8480fcd80 msgr2=0x7fb848105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:25.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 shutdown_connections 2024-03-18T15:59:25.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:25.896+0000 7fb84cd77640 1 -- 172.21.15.150:0/983089872 wait complete. 2024-03-18T15:59:25.908 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:25 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3469202804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.908 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:25 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2007968304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.908 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:25 smithi150 ceph-mon[31946]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:25.908 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:25 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/309025346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T15:59:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:25 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3469202804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:25 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2007968304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:25 smithi191 ceph-mon[36784]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:25 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/309025346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 -- 172.21.15.150:0/2671081989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 msgr2=0x7f0c1c101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 --2- 172.21.15.150:0/2671081989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c101d60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009900 tx=0x7f0c0c02bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 -- 172.21.15.150:0/2671081989 shutdown_connections 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 --2- 172.21.15.150:0/2671081989 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1022a0 0x7f0c1c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 --2- 172.21.15.150:0/2671081989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c101d60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 --2- 172.21.15.150:0/2671081989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1006f0 0x7f0c1c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 -- 172.21.15.150:0/2671081989 >> 172.21.15.150:0/2671081989 conn(0x7f0c1c0fbc80 msgr2=0x7f0c1c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 -- 172.21.15.150:0/2671081989 shutdown_connections 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.009+0000 7f0c247b3640 1 -- 172.21.15.150:0/2671081989 wait complete. 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 Processor -- start 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 -- start start 2024-03-18T15:59:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1006f0 0x7f0c1c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1022a0 0x7f0c1c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c370 con 0x7f0c1c1018e0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c1f0 con 0x7f0c1c1022a0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c247b3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c10c4f0 con 0x7f0c1c1006f0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51396/0 (socket says 172.21.15.150:51396) 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 -- 172.21.15.150:0/755145409 learned_addr learned my addr 172.21.15.150:0/755145409 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c22d50640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1022a0 0x7f0c1c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c2254f640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1006f0 0x7f0c1c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 -- 172.21.15.150:0/755145409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1006f0 msgr2=0x7f0c1c196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1006f0 0x7f0c1c196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 -- 172.21.15.150:0/755145409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1022a0 msgr2=0x7f0c1c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1022a0 0x7f0c1c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.010+0000 7f0c21d4e640 1 -- 172.21.15.150:0/755145409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c069dd0 con 0x7f0c1c1018e0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009230 tx=0x7f0c0c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c137fe640 1 -- 172.21.15.150:0/755145409 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c041070 con 0x7f0c1c1018e0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c137fe640 1 -- 172.21.15.150:0/755145409 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c004450 con 0x7f0c1c1018e0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c137fe640 1 -- 172.21.15.150:0/755145409 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c003e20 con 0x7f0c1c1018e0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c06a060 con 0x7f0c1c1018e0 2024-03-18T15:59:26.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.011+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c06a550 con 0x7f0c1c1018e0 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c137fe640 1 -- 172.21.15.150:0/755145409 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f0c0c044070 con 0x7f0c1c1018e0 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 Processor -- start 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 -- start start 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 0x5613919ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561391928c80 con 0x5613919c4bd0 2024-03-18T15:59:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5613918d1aa0 con 0x5613919ca730 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.013+0000 7f0c2739a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561391924d20 con 0x5613919c55c0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c137fe640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 0x7f0c080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51412/0 (socket says 172.21.15.150:51412) 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 0x5613919ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 -- 172.21.15.150:0/490651155 learned_addr learned my addr 172.21.15.150:0/490651155 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 0x7f0c080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 -- 172.21.15.150:0/490651155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 msgr2=0x5613919ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c137fe640 1 -- 172.21.15.150:0/755145409 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0c0c0f0620 con 0x7f0c1c1018e0 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 -- 172.21.15.150:0/490651155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 msgr2=0x5613919c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 -- 172.21.15.150:0/490651155 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613918f3ff0 con 0x5613919c55c0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c22d50640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 0x7f0c080a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5613919c5e20 tx=0x7f0c14009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c2254f640 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 0x5613919ca0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009bf0 tx=0x7f0c18011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=776b216ed71539d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c10ff9640 1 -- 172.21.15.150:0/490651155 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18004050 con 0x5613919c55c0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c10ff9640 1 -- 172.21.15.150:0/490651155 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c18005500 con 0x5613919c55c0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.014+0000 7f0c10ff9640 1 -- 172.21.15.150:0/490651155 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18004dd0 con 0x5613919c55c0 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- 172.21.15.150:0/490651155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 msgr2=0x5613919ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 0x5613919ca0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009bf0 tx=0x7f0c18011810 comp rx=0 tx=0).stop 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- 172.21.15.150:0/490651155 shutdown_connections 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c55c0 0x5613919ca0c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 --2- 172.21.15.150:0/490651155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c4bd0 0x5613919c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- 172.21.15.150:0/490651155 >> 172.21.15.150:0/490651155 conn(0x5613919c0140 msgr2=0x5613919c0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- 172.21.15.150:0/490651155 shutdown_connections 2024-03-18T15:59:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- 172.21.15.150:0/490651155 wait complete. 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 -- start start 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.015+0000 7f0c2739a740 1 Processor -- start 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 0x5613919cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5613919cd6e0 con 0x5613919ca730 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5613918e4ef0 con 0x5613919c55c0 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5613919cd860 con 0x5613919c4bd0 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 10 client.0 _ll_register_callbacks cb 0x5613918dfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2254f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51424/0 (socket says 172.21.15.150:51424) 2024-03-18T15:59:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 -- 172.21.15.150:0/3276004046 learned_addr learned my addr 172.21.15.150:0/3276004046 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 0x5613919cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 -- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 msgr2=0x5613919cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 -- 172.21.15.150:0/3276004046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 msgr2=0x5613919cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 0x5613919cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 -- 172.21.15.150:0/3276004046 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613918f1a30 con 0x5613919ca730 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2254f640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 0x5613919cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c22d50640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c14011fd0 tx=0x7f0c14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5c90c9c6d9961c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c140047c0 con 0x5613919ca730 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0c14004b20 con 0x5613919ca730 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.016+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c18005500 con 0x5613919ca730 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0c14004d20 con 0x5613919ca730 2024-03-18T15:59:26.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 10 client.15120 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56139163d2d0 con 0x5613919ca730 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c18005500 con 0x5613919ca730 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0c14028070 con 0x5613919ca730 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 10 client.15120 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 20 client.15120 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5613919db8d0 0x5613919dcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 4 client.15120 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613919361c0 con 0x5613919db8d0 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.017+0000 7f0c2254f640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5613919db8d0 0x5613919dcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c05ffb640 10 client.15120 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2254f640 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5613919db8d0 0x5613919dcd40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12fb9c1ebd9f5316 server_cookie=4fbc626e72d457d4 in_seq=0 out_seq=0 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c05ffb640 1 -- 172.21.15.150:0/3276004046 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613919361c0 con 0x5613919db8d0 2024-03-18T15:59:26.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c05ffb640 10 client.15120 handle_command_reply: tid=0 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 client.15120 shutdown 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 msgr2=0x5613919d6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c14011fd0 tx=0x7f0c14004a40 comp rx=0 tx=0).stop 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 shutdown_connections 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5613919db8d0 0x5613919dcd40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919ca730 0x5613919d6330 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c55c0 0x5613919cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 --2- 172.21.15.150:0/3276004046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 >> 172.21.15.150:0/3276004046 conn(0x5613919c0140 msgr2=0x5613919bf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c05ffb640 0 client.15120 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.018+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 shutdown_connections 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.019+0000 7f0c2739a740 1 -- 172.21.15.150:0/3276004046 wait complete. 2024-03-18T15:59:26.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.019+0000 7f0c2739a740 20 client.15120 trim_cache size 0 max 16384 2024-03-18T15:59:26.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:25 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3469202804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:26.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:25 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2007968304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:26.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:25 smithi156 ceph-mon[35919]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:26.029 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:25 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/309025346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 Processor -- start 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 -- start start 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5613919db080 con 0x5613919c55c0 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5613919db750 con 0x5613919ca730 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.036+0000 7f0c2739a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5613919dbe20 con 0x5613919c4bd0 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51426/0 (socket says 172.21.15.150:51426) 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 -- 172.21.15.150:0/764466870 learned_addr learned my addr 172.21.15.150:0/764466870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c2254f640 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c22d50640 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 -- 172.21.15.150:0/764466870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 msgr2=0x5613919cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 -- 172.21.15.150:0/764466870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 msgr2=0x5613919cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 -- 172.21.15.150:0/764466870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613917076b0 con 0x5613919c55c0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0bcd50 tx=0x7f0c0c08ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486cae048c77ebb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c07fff640 1 -- 172.21.15.150:0/764466870 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c04b040 con 0x5613919c55c0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c07fff640 1 -- 172.21.15.150:0/764466870 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c004c60 con 0x5613919c55c0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.037+0000 7f0c07fff640 1 -- 172.21.15.150:0/764466870 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0c52b0 con 0x5613919c55c0 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- 172.21.15.150:0/764466870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 msgr2=0x5613919d6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0bcd50 tx=0x7f0c0c08ecd0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- 172.21.15.150:0/764466870 shutdown_connections 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919ca730 0x5613919cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d6250 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- 172.21.15.150:0/764466870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919c4bd0 0x5613919cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- 172.21.15.150:0/764466870 >> 172.21.15.150:0/764466870 conn(0x5613919c0140 msgr2=0x5613919da670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- 172.21.15.150:0/764466870 shutdown_connections 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- 172.21.15.150:0/764466870 wait complete. 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- start start 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 Processor -- start 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 0x5613919d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919ca730 0x5613919d5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5613919dcbc0 con 0x5613919c55c0 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5613919dc4f0 con 0x5613919c4bd0 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5613919dd290 con 0x5613919ca730 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2739a740 10 client.0 _ll_register_callbacks cb 0x7f0c14011e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c2254f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 0x5613919d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c21d4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51430/0 (socket says 172.21.15.150:51430) 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c21d4e640 1 -- 172.21.15.150:0/75814927 learned_addr learned my addr 172.21.15.150:0/75814927 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.038+0000 7f0c22d50640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919ca730 0x5613919d5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 -- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919ca730 msgr2=0x5613919d5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919ca730 0x5613919d5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 -- 172.21.15.150:0/75814927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 msgr2=0x5613919d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 0x5613919d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 -- 172.21.15.150:0/75814927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613919d6560 con 0x5613919c55c0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2254f640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 0x5613919d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c21d4e640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0b94b0 tx=0x7f0c0c0bca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a3a34cbfdf0709 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0db060 con 0x5613919c55c0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0c0c0b7bc0 con 0x5613919c55c0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c0c004c60 con 0x5613919c55c0 2024-03-18T15:59:26.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0c0c04b070 con 0x5613919c55c0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 10 client.15126 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561391a40570 con 0x5613919c55c0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c004c60 con 0x5613919c55c0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f0c0c0bbf10 con 0x5613919c55c0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 10 client.15126 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 20 client.15126 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561391a44f20 0x561391a41a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 4 client.15126 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "status"}) -- 0x561391930510 con 0x561391a44f20 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.039+0000 7f0c2254f640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561391a44f20 0x561391a41a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c05ffb640 10 client.15126 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2254f640 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561391a44f20 0x561391a41a30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a39412530e3079f9 server_cookie=b37e3c2cfb4b3150 in_seq=0 out_seq=0 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c05ffb640 1 -- 172.21.15.150:0/75814927 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x561391930510 con 0x561391a44f20 2024-03-18T15:59:26.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c05ffb640 10 client.15126 handle_command_reply: tid=0 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 client.15126 shutdown 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 msgr2=0x5613919d0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0b94b0 tx=0x7f0c0c0bca70 comp rx=0 tx=0).stop 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 shutdown_connections 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561391a44f20 0x561391a41a30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5613919ca730 0x5613919d5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c05ffb640 0 client.15126 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5613919c55c0 0x5613919d0f70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 --2- 172.21.15.150:0/75814927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613919c4bd0 0x5613919d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 >> 172.21.15.150:0/75814927 conn(0x5613919c0140 msgr2=0x5613919da670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 shutdown_connections 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 1 -- 172.21.15.150:0/75814927 wait complete. 2024-03-18T15:59:26.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.040+0000 7f0c2739a740 20 client.15126 trim_cache size 0 max 16384 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24496, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 88.882850399000006, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 103.58543883 2024-03-18T15:59:26.051 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:26.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 msgr2=0x7f0c080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 0x7f0c080a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5613919c5e20 tx=0x7f0c14009500 comp rx=0 tx=0).stop 2024-03-18T15:59:26.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 msgr2=0x7f0c1c196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009230 tx=0x7f0c0c02c650 comp rx=0 tx=0).stop 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 shutdown_connections 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0c080a2890 0x7f0c080a4d50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c1c1022a0 0x7f0c1c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c1c1018e0 0x7f0c1c196f40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 --2- 172.21.15.150:0/755145409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c1c1006f0 0x7f0c1c196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 >> 172.21.15.150:0/755145409 conn(0x7f0c1c0fbc80 msgr2=0x7f0c1c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 shutdown_connections 2024-03-18T15:59:26.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:26.041+0000 7f0c247b3640 1 -- 172.21.15.150:0/755145409 wait complete. 2024-03-18T15:59:26.054 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T15:59:26.054 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T15:59:26.064 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T15:59:26.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T15:59:26.305 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:27.769 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:27 smithi150 ceph-mon[31946]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T15:59:27.770 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:27 smithi150 ceph-mon[31946]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:27 smithi191 ceph-mon[36784]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T15:59:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:27 smithi191 ceph-mon[36784]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 -- 172.21.15.150:0/3332337711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102400 msgr2=0x7f7d34102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:28.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3332337711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102400 0x7f7d34102860 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7d240099c0 tx=0x7f7d24009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 -- 172.21.15.150:0/3332337711 shutdown_connections 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3332337711 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34102da0 0x7f7d34109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3332337711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34101200 0x7f7d34101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3332337711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102400 0x7f7d34102860 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 -- 172.21.15.150:0/3332337711 >> 172.21.15.150:0/3332337711 conn(0x7f7d340fc7c0 msgr2=0x7f7d340febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 -- 172.21.15.150:0/3332337711 shutdown_connections 2024-03-18T15:59:28.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.000+0000 7f7d3a416700 1 -- 172.21.15.150:0/3332337711 wait complete. 2024-03-18T15:59:28.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 Processor -- start 2024-03-18T15:59:28.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 -- start start 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34102400 0x7f7d34198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102da0 0x7f7d3419fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410ce00 con 0x7f7d34102da0 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410cc80 con 0x7f7d34102400 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d3a416700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410cf80 con 0x7f7d34101200 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d389b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102da0 0x7f7d3419fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:50882/0 (socket says 172.21.15.150:50882) 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 -- 172.21.15.150:0/3111134061 learned_addr learned my addr 172.21.15.150:0/3111134061 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d337fe700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34102400 0x7f7d34198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 -- 172.21.15.150:0/3111134061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34102400 msgr2=0x7f7d34198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34102400 0x7f7d34198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 -- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102da0 msgr2=0x7f7d3419fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:28.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102da0 0x7f7d3419fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.001+0000 7f7d33fff700 1 -- 172.21.15.150:0/3111134061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d341a01c0 con 0x7f7d34101200 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d33fff700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800e9a0 tx=0x7f7d2800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28004220 con 0x7f7d34101200 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d341a04b0 con 0x7f7d34101200 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d280043c0 con 0x7f7d34101200 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28003df0 con 0x7f7d34101200 2024-03-18T15:59:28.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.002+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d341a09f0 con 0x7f7d34101200 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.003+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d34068f90 con 0x7f7d34101200 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.004+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7d2800c810 con 0x7f7d34101200 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.004+0000 7f7d317fa700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 0x7f7d1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.005+0000 7f7d337fe700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 0x7f7d1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.005+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7d280cebd0 con 0x7f7d34101200 2024-03-18T15:59:28.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.005+0000 7f7d337fe700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 0x7f7d1c0a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000c00 tx=0x7f7d24005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.014+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d2809bca0 con 0x7f7d34101200 2024-03-18T15:59:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:27 smithi156 ceph-mon[35919]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T15:59:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:27 smithi156 ceph-mon[35919]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:28.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.244+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f7d340fe930 con 0x7f7d1c0a2890 2024-03-18T15:59:28.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.245+0000 7f7d317fa700 1 -- 172.21.15.150:0/3111134061 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f7d340fe930 con 0x7f7d1c0a2890 2024-03-18T15:59:28.248 INFO:teuthology.orchestra.run.smithi150.stdout:Backend: cephadm 2024-03-18T15:59:28.248 INFO:teuthology.orchestra.run.smithi150.stdout:Available: Yes 2024-03-18T15:59:28.248 INFO:teuthology.orchestra.run.smithi150.stdout:Paused: No 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.247+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 msgr2=0x7f7d1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 0x7f7d1c0a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000c00 tx=0x7f7d24005fb0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 msgr2=0x7f7d34198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800e9a0 tx=0x7f7d2800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 shutdown_connections 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34102da0 0x7f7d3419fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d34101200 0x7f7d34198560 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7d1c0a2890 0x7f7d1c0a4d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 --2- 172.21.15.150:0/3111134061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34102400 0x7f7d34198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:28.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 >> 172.21.15.150:0/3111134061 conn(0x7f7d340fc7c0 msgr2=0x7f7d340fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:28.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 shutdown_connections 2024-03-18T15:59:28.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:28.248+0000 7f7d3a416700 1 -- 172.21.15.150:0/3111134061 wait complete. 2024-03-18T15:59:28.890 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:28 smithi150 ceph-mon[31946]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:28.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T15:59:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:28 smithi191 ceph-mon[36784]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:29.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:28 smithi156 ceph-mon[35919]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:29.162 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:29 smithi150 ceph-mon[31946]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:29 smithi191 ceph-mon[36784]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:29 smithi156 ceph-mon[35919]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:30.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.972+0000 7f7edc75b700 1 -- 172.21.15.150:0/2926458095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed41034f0 msgr2=0x7f7ed4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:30.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.972+0000 7f7edc75b700 1 --2- 172.21.15.150:0/2926458095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed41034f0 0x7f7ed4103950 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00099c0 tx=0x7f7ed0009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- 172.21.15.150:0/2926458095 shutdown_connections 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- 172.21.15.150:0/2926458095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed4103e90 0x7f7ed410a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- 172.21.15.150:0/2926458095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 0x7f7ed4102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- 172.21.15.150:0/2926458095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed41034f0 0x7f7ed4103950 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- 172.21.15.150:0/2926458095 >> 172.21.15.150:0/2926458095 conn(0x7f7ed40fd8b0 msgr2=0x7f7ed40ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- 172.21.15.150:0/2926458095 shutdown_connections 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- 172.21.15.150:0/2926458095 wait complete. 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 Processor -- start 2024-03-18T15:59:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- start start 2024-03-18T15:59:30.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 0x7f7ed41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:30.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 0x7f7ed4198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:30.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:30.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410dee0 con 0x7f7ed4103e90 2024-03-18T15:59:30.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410dd60 con 0x7f7ed41034f0 2024-03-18T15:59:30.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edc75b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410e060 con 0x7f7ed41022f0 2024-03-18T15:59:30.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edacf8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7edacf8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51466/0 (socket says 172.21.15.150:51466) 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.973+0000 7f7ed9cf6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 0x7f7ed4198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 -- 172.21.15.150:0/3689152673 learned_addr learned my addr 172.21.15.150:0/3689152673 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7eda4f7700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 0x7f7ed41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 -- 172.21.15.150:0/3689152673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 msgr2=0x7f7ed41984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 0x7f7ed41984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 -- 172.21.15.150:0/3689152673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 msgr2=0x7f7ed4198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 0x7f7ed4198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 -- 172.21.15.150:0/3689152673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed41a0150 con 0x7f7ed4103e90 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7ed9cf6700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 0x7f7ed4198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edacf8700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00ef60 tx=0x7f7ecc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:30.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0092e0 con 0x7f7ed4103e90 2024-03-18T15:59:30.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ecc004bb0 con 0x7f7ed4103e90 2024-03-18T15:59:30.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc004280 con 0x7f7ed4103e90 2024-03-18T15:59:30.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed41a03e0 con 0x7f7ed4103e90 2024-03-18T15:59:30.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.974+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed41a0970 con 0x7f7ed4103e90 2024-03-18T15:59:30.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.975+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed4068f90 con 0x7f7ed4103e90 2024-03-18T15:59:30.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.976+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7ecc01b030 con 0x7f7ed4103e90 2024-03-18T15:59:30.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.977+0000 7f7ecb7fe700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 0x7f7ec00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:30.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.977+0000 7f7eda4f7700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 0x7f7ec00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:30.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.977+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7ecc0ce3c0 con 0x7f7ed4103e90 2024-03-18T15:59:30.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.977+0000 7f7eda4f7700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 0x7f7ec00a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4103350 tx=0x7f7ec4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:30.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:30.981+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ecc022020 con 0x7f7ed4103e90 2024-03-18T15:59:31.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.184+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f7ed4062710 con 0x7f7ec00a2910 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.198+0000 7f7ecb7fe700 1 -- 172.21.15.150:0/3689152673 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f7ed4062710 con 0x7f7ec00a2910 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.a smithi150 running (103s) 18s ago 102s 13.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 8e509fcc0158 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.b smithi156 running (100s) 34s ago 99s 20.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 03004a5dc058 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.c smithi191 running (114s) 34s ago 114s 16.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4608e976db80 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.d smithi150 running (2m) 18s ago 2m 14.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 033a22e8745d 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.e smithi156 running (2m) 34s ago 2m 15.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 412fb943d569 2024-03-18T15:59:31.200 INFO:teuthology.orchestra.run.smithi150.stdout:mds.f smithi191 running (117s) 34s ago 117s 17.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 908b684a7ed2 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.g smithi150 running (106s) 18s ago 105s 13.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ddd2d6b72776 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.h smithi156 running (112s) 34s ago 111s 16.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 27d513d08cb8 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.i smithi191 running (109s) 34s ago 108s 28.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ad5640ce5aab 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.j smithi150 running (2m) 18s ago 2m 16.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 828b49adc05d 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.k smithi156 running (2m) 34s ago 2m 14.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 85c0e1a37570 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mds.l smithi191 running (2m) 34s ago 2m 19.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b1298fd4c56e 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mgr.x smithi150 *:9283,8765 running (9m) 18s ago 9m 513M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 5a0d9a31a9ca 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mgr.y smithi156 *:8443,8765 running (7m) 34s ago 7m 451M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 320f6005c9ae 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mgr.z smithi191 *:8443,8765 running (7m) 34s ago 7m 454M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b8274b560dec 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mon.a smithi150 running (9m) 18s ago 9m 52.3M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 71bcd2f54d31 2024-03-18T15:59:31.201 INFO:teuthology.orchestra.run.smithi150.stdout:mon.b smithi156 running (7m) 34s ago 7m 40.3M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b e907da403e8b 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:mon.c smithi191 running (7m) 34s ago 7m 42.4M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 32f6bc5cc7bb 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0 smithi150 running (7m) 18s ago 7m 55.8M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 23424c99f231 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1 smithi156 running (6m) 34s ago 6m 59.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 458b9284a3df 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2 smithi191 running (6m) 34s ago 6m 59.5M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 6b00e23ef373 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3 smithi150 running (5m) 18s ago 5m 57.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 1433e380836b 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4 smithi156 running (5m) 34s ago 5m 41.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2326ab523bc7 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5 smithi191 running (5m) 34s ago 5m 54.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b d807845a890f 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6 smithi150 running (4m) 18s ago 4m 42.5M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 7a7e84cad1d3 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7 smithi156 running (4m) 34s ago 4m 54.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8196aefa8669 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.8 smithi191 running (3m) 34s ago 3m 54.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b c25666d668d4 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.9 smithi150 running (3m) 18s ago 3m 56.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b dde576b5a280 2024-03-18T15:59:31.202 INFO:teuthology.orchestra.run.smithi150.stdout:osd.10 smithi156 running (2m) 34s ago 2m 43.1M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b cac2552c94fc 2024-03-18T15:59:31.203 INFO:teuthology.orchestra.run.smithi150.stdout:osd.11 smithi191 running (2m) 34s ago 2m 42.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b aade0d9cc405 2024-03-18T15:59:31.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 msgr2=0x7f7ec00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:31.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 0x7f7ec00a4dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4103350 tx=0x7f7ec4009260 comp rx=0 tx=0).stop 2024-03-18T15:59:31.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 msgr2=0x7f7ed419fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00ef60 tx=0x7f7ecc00ef90 comp rx=0 tx=0).stop 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 shutdown_connections 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed4103e90 0x7f7ed419fab0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ed41022f0 0x7f7ed41984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7ec00a2910 0x7f7ec00a4dd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 --2- 172.21.15.150:0/3689152673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ed41034f0 0x7f7ed4198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.200+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 >> 172.21.15.150:0/3689152673 conn(0x7f7ed40fd8b0 msgr2=0x7f7ed4106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.201+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 shutdown_connections 2024-03-18T15:59:31.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:31.201+0000 7f7edc75b700 1 -- 172.21.15.150:0/3689152673 wait complete. 2024-03-18T15:59:31.969 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T15:59:32.221 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:32.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:32 smithi150 ceph-mon[31946]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:32.673 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:32 smithi150 ceph-mon[31946]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:32.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:32 smithi191 ceph-mon[36784]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:32.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:32 smithi191 ceph-mon[36784]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:33.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:32 smithi156 ceph-mon[35919]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:33.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:32 smithi156 ceph-mon[35919]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:34.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 -- 172.21.15.150:0/3587058403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30102400 msgr2=0x7ffb30102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 --2- 172.21.15.150:0/3587058403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30102400 0x7ffb30102860 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb180099c0 tx=0x7ffb18009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 -- 172.21.15.150:0/3587058403 shutdown_connections 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 --2- 172.21.15.150:0/3587058403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 0x7ffb30109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 --2- 172.21.15.150:0/3587058403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30101200 0x7ffb30101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 --2- 172.21.15.150:0/3587058403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30102400 0x7ffb30102860 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 -- 172.21.15.150:0/3587058403 >> 172.21.15.150:0/3587058403 conn(0x7ffb300fc7a0 msgr2=0x7ffb300febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 -- 172.21.15.150:0/3587058403 shutdown_connections 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.010+0000 7ffb36146700 1 -- 172.21.15.150:0/3587058403 wait complete. 2024-03-18T15:59:34.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 Processor -- start 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 -- start start 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 0x7ffb301037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 0x7ffb30103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010ce30 con 0x7ffb30101200 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010ccb0 con 0x7ffb30102400 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb36146700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3010cfb0 con 0x7ffb30102da0 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb2f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb2f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34508/0 (socket says 172.21.15.150:34508) 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb2f7fe700 1 -- 172.21.15.150:0/4288817999 learned_addr learned my addr 172.21.15.150:0/4288817999 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.011+0000 7ffb26ffd700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 0x7ffb301037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb2ffff700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 0x7ffb30103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 -- 172.21.15.150:0/4288817999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 msgr2=0x7ffb30103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 0x7ffb30103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 -- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 msgr2=0x7ffb30105190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 -- 172.21.15.150:0/4288817999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb301045d0 con 0x7ffb30102400 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb2f7fe700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb26ffd700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 0x7ffb301037e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009990 tx=0x7ffb1800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18009e60 con 0x7ffb30102400 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb18003960 con 0x7ffb30102400 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb301a26a0 con 0x7ffb30102400 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18005640 con 0x7ffb30102400 2024-03-18T15:59:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.012+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb301a2af0 con 0x7ffb30102400 2024-03-18T15:59:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.013+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30068f90 con 0x7ffb30102400 2024-03-18T15:59:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.014+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ffb18003b10 con 0x7ffb30102400 2024-03-18T15:59:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.015+0000 7ffb2d7fa700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 0x7ffb1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.015+0000 7ffb2f7fe700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 0x7ffb1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.015+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ffb180cf610 con 0x7ffb30102400 2024-03-18T15:59:34.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.015+0000 7ffb2f7fe700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 0x7ffb1c0a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb20005950 tx=0x7ffb20009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.020+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb18013070 con 0x7ffb30102400 2024-03-18T15:59:34.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.226+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ffb300fd530 con 0x7ffb1c0a2890 2024-03-18T15:59:34.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.232+0000 7ffb2d7fa700 1 -- 172.21.15.150:0/4288817999 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ffb300fd530 con 0x7ffb1c0a2890 2024-03-18T15:59:34.233 INFO:teuthology.orchestra.run.smithi150.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T15:59:34.233 INFO:teuthology.orchestra.run.smithi150.stdout:mds.all 12/12 37s ago 2m smithi150=a;smithi150=d;smithi150=g;smithi150=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2024-03-18T15:59:34.233 INFO:teuthology.orchestra.run.smithi150.stdout:mgr 3/3 37s ago 7m smithi150=x;smithi156=y;smithi191=z;count:3 2024-03-18T15:59:34.234 INFO:teuthology.orchestra.run.smithi150.stdout:mon 3/3 37s ago 8m smithi150:172.21.15.150=a;smithi156:172.21.15.156=b;smithi191:172.21.15.191=c;count:3 2024-03-18T15:59:34.234 INFO:teuthology.orchestra.run.smithi150.stdout:osd 12 37s ago - 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 msgr2=0x7ffb1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 0x7ffb1c0a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb20005950 tx=0x7ffb20009260 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 msgr2=0x7ffb301037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 0x7ffb301037e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009990 tx=0x7ffb1800f690 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 shutdown_connections 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb30102da0 0x7ffb30103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb30101200 0x7ffb30105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ffb1c0a2890 0x7ffb1c0a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.234+0000 7ffb36146700 1 --2- 172.21.15.150:0/4288817999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb30102400 0x7ffb301037e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.235+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 >> 172.21.15.150:0/4288817999 conn(0x7ffb300fc7a0 msgr2=0x7ffb300fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:34.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.235+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 shutdown_connections 2024-03-18T15:59:34.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:34.235+0000 7ffb36146700 1 -- 172.21.15.150:0/4288817999 wait complete. 2024-03-18T15:59:34.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:34 smithi150 ceph-mon[31946]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:34.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T15:59:34.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:34 smithi191 ceph-mon[36784]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:35.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:34 smithi156 ceph-mon[35919]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:35.185 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:35.566 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:35 smithi150 ceph-mon[31946]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:35 smithi191 ceph-mon[36784]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:35 smithi156 ceph-mon[35919]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:36.607 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:36 smithi150 ceph-mon[31946]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:36 smithi191 ceph-mon[36784]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 -- 172.21.15.150:0/2244425040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102400 msgr2=0x7fe12c102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2244425040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102400 0x7fe12c102860 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe1140099c0 tx=0x7fe114009cd0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 -- 172.21.15.150:0/2244425040 shutdown_connections 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2244425040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c102da0 0x7fe12c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2244425040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c101200 0x7fe12c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2244425040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102400 0x7fe12c102860 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 -- 172.21.15.150:0/2244425040 >> 172.21.15.150:0/2244425040 conn(0x7fe12c0fc7c0 msgr2=0x7fe12c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 -- 172.21.15.150:0/2244425040 shutdown_connections 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.986+0000 7fe1318e2700 1 -- 172.21.15.150:0/2244425040 wait complete. 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 Processor -- start 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 -- start start 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c102400 0x7fe12c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 0x7fe12c19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:37.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c10ce00 con 0x7fe12c102da0 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c10cc80 con 0x7fe12c102400 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe1318e2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c10cf80 con 0x7fe12c101200 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 0x7fe12c19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12a7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c102400 0x7fe12c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:41218/0 (socket says 172.21.15.150:41218) 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12affd700 1 -- 172.21.15.150:0/2791020848 learned_addr learned my addr 172.21.15.150:0/2791020848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.987+0000 7fe12b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 0x7fe12c19fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34526/0 (socket says 172.21.15.150:34526) 2024-03-18T15:59:37.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 -- 172.21.15.150:0/2791020848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c102400 msgr2=0x7fe12c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c102400 0x7fe12c198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 -- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 msgr2=0x7fe12c19fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 0x7fe12c19fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 -- 172.21.15.150:0/2791020848 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c1a01a0 con 0x7fe12c101200 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe12affd700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00e9a0 tx=0x7fe11c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c004220 con 0x7fe12c101200 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12c1a0490 con 0x7fe12c101200 2024-03-18T15:59:37.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe11c0043c0 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c003df0 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.988+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12c1a0a20 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.989+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe12c068f90 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.990+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe11c00c810 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.990+0000 7fe1308e0700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 0x7fe1180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.991+0000 7fe12a7fc700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 0x7fe1180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.991+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe11c0cec50 con 0x7fe12c101200 2024-03-18T15:59:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.991+0000 7fe12a7fc700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 0x7fe1180a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe114000c00 tx=0x7fe114005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:36.999+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe11c09bcc0 con 0x7fe12c101200 2024-03-18T15:59:37.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:36 smithi156 ceph-mon[35919]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:37.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.198+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe12c06b100 con 0x7fe1180a2910 2024-03-18T15:59:37.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.200+0000 7fe1308e0700 1 -- 172.21.15.150:0/2791020848 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe12c06b100 con 0x7fe1180a2910 2024-03-18T15:59:37.201 INFO:teuthology.orchestra.run.smithi150.stdout:HOST ADDR LABELS STATUS 2024-03-18T15:59:37.201 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150 172.21.15.150 2024-03-18T15:59:37.201 INFO:teuthology.orchestra.run.smithi150.stdout:smithi156 172.21.15.156 2024-03-18T15:59:37.202 INFO:teuthology.orchestra.run.smithi150.stdout:smithi191 172.21.15.191 2024-03-18T15:59:37.202 INFO:teuthology.orchestra.run.smithi150.stdout:3 hosts in cluster 2024-03-18T15:59:37.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 msgr2=0x7fe1180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 0x7fe1180a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe114000c00 tx=0x7fe114005fb0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 msgr2=0x7fe12c198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00e9a0 tx=0x7fe11c00ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 shutdown_connections 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c102da0 0x7fe12c19fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe12c101200 0x7fe12c198540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe1180a2910 0x7fe1180a4dd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 --2- 172.21.15.150:0/2791020848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe12c102400 0x7fe12c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 >> 172.21.15.150:0/2791020848 conn(0x7fe12c0fc7c0 msgr2=0x7fe12c06a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 shutdown_connections 2024-03-18T15:59:37.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:37.202+0000 7fe1318e2700 1 -- 172.21.15.150:0/2791020848 wait complete. 2024-03-18T15:59:37.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T15:59:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:38.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:38 smithi150 ceph-mon[31946]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:38.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:38 smithi150 ceph-mon[31946]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:38.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:38 smithi191 ceph-mon[36784]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:38.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:38 smithi191 ceph-mon[36784]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:38 smithi156 ceph-mon[35919]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:39.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:38 smithi156 ceph-mon[35919]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-18T15:59:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- 172.21.15.150:0/3826916444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c10a1e0 msgr2=0x7f281c10c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 --2- 172.21.15.150:0/3826916444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c10a1e0 0x7f281c10c670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f281400b0a0 tx=0x7f281400b3b0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- 172.21.15.150:0/3826916444 shutdown_connections 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 --2- 172.21.15.150:0/3826916444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c10a1e0 0x7f281c10c670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 --2- 172.21.15.150:0/3826916444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c0762a0 0x7f281c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 --2- 172.21.15.150:0/3826916444 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c075150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- 172.21.15.150:0/3826916444 >> 172.21.15.150:0/3826916444 conn(0x7f281c0fd8e0 msgr2=0x7f281c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- 172.21.15.150:0/3826916444 shutdown_connections 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- 172.21.15.150:0/3826916444 wait complete. 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 Processor -- start 2024-03-18T15:59:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.878+0000 7f2821454700 1 -- start start 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c10a1e0 0x7f281c1a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1100c0 con 0x7f281c0762a0 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f281c10ff40 con 0x7f281c10a1e0 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f281c110240 con 0x7f281c074cd0 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281affd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34546/0 (socket says 172.21.15.150:34546) 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 -- 172.21.15.150:0/1879896899 learned_addr learned my addr 172.21.15.150:0/1879896899 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281b7fe700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c10a1e0 0x7f281c1a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 -- 172.21.15.150:0/1879896899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 msgr2=0x7f281c19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 -- 172.21.15.150:0/1879896899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c10a1e0 msgr2=0x7f281c1a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c10a1e0 0x7f281c1a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 -- 172.21.15.150:0/1879896899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c1a22e0 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281affd700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f281a7fc700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f281000e9f0 tx=0x7f281000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281000cbd0 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c1a25d0 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f281000ee90 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28100056c0 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.879+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c1a2b10 con 0x7f281c0762a0 2024-03-18T15:59:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.880+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f281c068f90 con 0x7f281c0762a0 2024-03-18T15:59:39.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.881+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2810004750 con 0x7f281c0762a0 2024-03-18T15:59:39.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.882+0000 7f2803fff700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 0x7f28040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.882+0000 7f281affd700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 0x7f28040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.882+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f28100cec70 con 0x7f281c0762a0 2024-03-18T15:59:39.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.883+0000 7f281affd700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 0x7f28040a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f280c011490 tx=0x7f280c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:39.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:39.886+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f281009bd40 con 0x7f281c0762a0 2024-03-18T15:59:40.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.092+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f281c062710 con 0x7f28040a2890 2024-03-18T15:59:40.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.094+0000 7f2803fff700 1 -- 172.21.15.150:0/1879896899 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f281c062710 con 0x7f28040a2890 2024-03-18T15:59:40.095 INFO:teuthology.orchestra.run.smithi150.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T15:59:40.096 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T15:59:40.096 INFO:teuthology.orchestra.run.smithi150.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-18T15:59:40.096 INFO:teuthology.orchestra.run.smithi150.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 msgr2=0x7f28040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 0x7f28040a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f280c011490 tx=0x7f280c009260 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 msgr2=0x7f281c19abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f281000e9f0 tx=0x7f281000ed00 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 shutdown_connections 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f28040a2890 0x7f28040a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f281c10a1e0 0x7f281c1a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f281c074cd0 0x7f281c19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 --2- 172.21.15.150:0/1879896899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f281c0762a0 0x7f281c19abc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 >> 172.21.15.150:0/1879896899 conn(0x7f281c0fd8e0 msgr2=0x7f281c106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 shutdown_connections 2024-03-18T15:59:40.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:40.096+0000 7f2821454700 1 -- 172.21.15.150:0/1879896899 wait complete. 2024-03-18T15:59:40.778 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T15:59:40.790 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T15:59:40.790 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T15:59:40.816 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:40 smithi150 ceph-mon[31946]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:40.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:40 smithi191 ceph-mon[36784]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:41.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:40 smithi156 ceph-mon[35919]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T15:59:41.033 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:41.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:41 smithi150 ceph-mon[31946]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:41 smithi191 ceph-mon[36784]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:41 smithi156 ceph-mon[35919]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:42.700 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:42 smithi150 ceph-mon[31946]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 -- 172.21.15.150:0/1320823503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 msgr2=0x7f718810a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 --2- 172.21.15.150:0/1320823503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718810a460 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7178009990 tx=0x7f7178009ca0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 -- 172.21.15.150:0/1320823503 shutdown_connections 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 --2- 172.21.15.150:0/1320823503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718810a460 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 --2- 172.21.15.150:0/1320823503 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 --2- 172.21.15.150:0/1320823503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 0x7f7188107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 -- 172.21.15.150:0/1320823503 >> 172.21.15.150:0/1320823503 conn(0x7f71880fc7a0 msgr2=0x7f71880febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:42.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 -- 172.21.15.150:0/1320823503 shutdown_connections 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 -- 172.21.15.150:0/1320823503 wait complete. 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.699+0000 7f718d548700 1 Processor -- start 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 -- start start 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 0x7f7188198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f718810df60 con 0x7f7188107ff0 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f718810dde0 con 0x7f71881056c0 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f718d548700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f718810e0e0 con 0x7f7188102d90 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f7186ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f7186ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:43646/0 (socket says 172.21.15.150:43646) 2024-03-18T15:59:42.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48092/0 (socket says 172.21.15.150:48092) 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f7186ffd700 1 -- 172.21.15.150:0/1651545549 learned_addr learned my addr 172.21.15.150:0/1651545549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71867fc700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 0x7f7188198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 -- 172.21.15.150:0/1651545549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 msgr2=0x7f7188198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 -- 172.21.15.150:0/1651545549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 msgr2=0x7f7188198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 0x7f7188198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f71877fe700 1 -- 172.21.15.150:0/1651545549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71881a01a0 con 0x7f7188107ff0 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.700+0000 7f7186ffd700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f71877fe700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f718806b060 tx=0x7f717800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178003890 con 0x7f7188107ff0 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71881a0430 con 0x7f7188107ff0 2024-03-18T15:59:42.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7178004830 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71881a0880 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.701+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178005370 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.702+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7188068f90 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.703+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7178009d10 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.703+0000 7f716ffff700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 0x7f71700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.703+0000 7f7186ffd700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 0x7f71700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.703+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f71780cfad0 con 0x7f7188107ff0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.704+0000 7f7186ffd700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 0x7f71700a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7180009c10 tx=0x7f718000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:42.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.707+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7178013070 con 0x7f7188107ff0 2024-03-18T15:59:42.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.944+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f718802db70 con 0x7f7188107ff0 2024-03-18T15:59:42.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.945+0000 7f716ffff700 1 -- 172.21.15.150:0/1651545549 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3060 (secure 0 0 0) 0x7f717809cc30 con 0x7f7188107ff0 2024-03-18T15:59:42.946 INFO:teuthology.orchestra.run.smithi150.stdout:e21 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:legacy client fscid: 1 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:Filesystem 'cephfs' (1) 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:fs_name cephfs 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 21 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-03-18T15:57:56.512066+0000 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:modified 2024-03-18T15:58:01.194272+0000 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:tableserver 0 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:root 0 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:session_timeout 300 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:session_autoclose 300 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:max_file_size 1099511627776 2024-03-18T15:59:42.947 INFO:teuthology.orchestra.run.smithi150.stdout:max_xattr_size 65536 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:required_client_features {} 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:last_failure 0 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:last_failure_osd_epoch 0 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:max_mds 3 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:in 0,1,2 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:up {0=24496,1=24485,2=24481} 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:failed 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:damaged 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:stopped 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:data_pools [3] 2024-03-18T15:59:42.948 INFO:teuthology.orchestra.run.smithi150.stdout:metadata_pool 2 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:inline_data disabled 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:balancer 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:bal_rank_mask -1 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:standby_count_wanted 0 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:qdb_cluster leader: 24496 members: 24481,24485,24496 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.i{0:24496} state up:active seq 5 addr [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.f{0:24463} state up:standby-replay seq 1 addr [v2:172.21.15.191:6834/2169559605,v1:172.21.15.191:6835/2169559605] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b{1:24485} state up:active seq 4 addr [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.h{1:24458} state up:standby-replay seq 1 addr [v2:172.21.15.156:6836/1894652569,v1:172.21.15.156:6837/1894652569] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c{2:24481} state up:active seq 7 addr [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.l{2:24430} state up:standby-replay seq 1 addr [v2:172.21.15.191:6832/2692248421,v1:172.21.15.191:6833/2692248421] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:42.949 INFO:teuthology.orchestra.run.smithi150.stdout:Standby daemons: 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.j{-1:14577} state up:standby seq 1 addr [v2:172.21.15.150:6834/3457359826,v1:172.21.15.150:6835/3457359826] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d{-1:14589} state up:standby seq 1 addr [v2:172.21.15.150:6836/3172588678,v1:172.21.15.150:6837/3172588678] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.g{-1:14649} state up:standby seq 1 addr [v2:172.21.15.150:6838/4258774828,v1:172.21.15.150:6839/4258774828] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a{-1:14667} state up:standby seq 1 addr [v2:172.21.15.150:6840/4215738622,v1:172.21.15.150:6841/4215738622] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.k{-1:24392} state up:standby seq 1 addr [v2:172.21.15.156:6832/925824611,v1:172.21.15.156:6833/925824611] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.950 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.e{-1:24422} state up:standby seq 1 addr [v2:172.21.15.156:6834/4268627742,v1:172.21.15.156:6835/4268627742] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 msgr2=0x7f71700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 0x7f71700a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7180009c10 tx=0x7f718000b040 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 msgr2=0x7f718819fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f718806b060 tx=0x7f717800bbb0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 shutdown_connections 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7188107ff0 0x7f718819fb00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7188102d90 0x7f7188198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71700a2890 0x7f71700a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 --2- 172.21.15.150:0/1651545549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71881056c0 0x7f7188198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:42.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 >> 172.21.15.150:0/1651545549 conn(0x7f71880fc7a0 msgr2=0x7f71881036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:42.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 shutdown_connections 2024-03-18T15:59:42.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:42.947+0000 7f718d548700 1 -- 172.21.15.150:0/1651545549 wait complete. 2024-03-18T15:59:42.952 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:42.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:42 smithi191 ceph-mon[36784]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:43.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:42 smithi156 ceph-mon[35919]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-18T15:59:43.833 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:43 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1651545549' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:59:43.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T15:59:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:43 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1651545549' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:59:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:43 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1651545549' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T15:59:44.149 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:44.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:44 smithi150 ceph-mon[31946]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:44.919 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:44 smithi156 ceph-mon[35919]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:44.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:44 smithi191 ceph-mon[36784]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:45.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- 172.21.15.150:0/832238232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 msgr2=0x7f6a9410a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:45.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 --2- 172.21.15.150:0/832238232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9410a770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00b0d0 tx=0x7f6a8c00b3e0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- 172.21.15.150:0/832238232 shutdown_connections 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 --2- 172.21.15.150:0/832238232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9410a770 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 --2- 172.21.15.150:0/832238232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a94102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 --2- 172.21.15.150:0/832238232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a94103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- 172.21.15.150:0/832238232 >> 172.21.15.150:0/832238232 conn(0x7f6a940fd8e0 msgr2=0x7f6a940ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- 172.21.15.150:0/832238232 shutdown_connections 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- 172.21.15.150:0/832238232 wait complete. 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 Processor -- start 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.956+0000 7f6a9c638700 1 -- start start 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a941984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a941989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410df60 con 0x7f6a94103ec0 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410dde0 con 0x7f6a94103520 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410e0e0 con 0x7f6a94102320 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48108/0 (socket says 172.21.15.150:48108) 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a99bd3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a941989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9a3d4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a941984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 -- 172.21.15.150:0/3983389543 learned_addr learned my addr 172.21.15.150:0/3983389543 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 -- 172.21.15.150:0/3983389543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 msgr2=0x7f6a941984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a941984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 -- 172.21.15.150:0/3983389543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 msgr2=0x7f6a941989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a941989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 -- 172.21.15.150:0/3983389543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a941a0110 con 0x7f6a94103ec0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a99bd3700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a941989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9a3d4700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a941984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9abd5700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c007fa0 tx=0x7f6a8c0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c003c00 con 0x7f6a94103ec0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c003da0 con 0x7f6a94103ec0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a941a03a0 con 0x7f6a94103ec0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c01b890 con 0x7f6a94103ec0 2024-03-18T15:59:45.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.957+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a941a0850 con 0x7f6a94103ec0 2024-03-18T15:59:45.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.958+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a94068f90 con 0x7f6a94103ec0 2024-03-18T15:59:45.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.960+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6a8c01ba30 con 0x7f6a94103ec0 2024-03-18T15:59:45.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.960+0000 7f6a8b7fe700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 0x7f6a800a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.960+0000 7f6a9a3d4700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 0x7f6a800a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.960+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6a8c00f020 con 0x7f6a94103ec0 2024-03-18T15:59:45.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.961+0000 7f6a9a3d4700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 0x7f6a800a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a94103380 tx=0x7f6a90009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:45.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:45.964+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a8c01a3c0 con 0x7f6a94103ec0 2024-03-18T15:59:46.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.163+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f6a9402dba0 con 0x7f6a94103ec0 2024-03-18T15:59:46.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.164+0000 7f6a8b7fe700 1 -- 172.21.15.150:0/3983389543 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5282 (secure 0 0 0) 0x7f6a8c09cc70 con 0x7f6a94103ec0 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 88 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-03-18T15:49:59.320410+0000 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:modified 2024-03-18T15:58:56.565844+0000 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:crush_version 26 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:full_ratio 0.95 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:backfillfull_ratio 0.9 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:nearfull_ratio 0.85 2024-03-18T15:59:46.166 INFO:teuthology.orchestra.run.smithi150.stdout:require_min_compat_client luminous 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:min_compat_client luminous 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:require_osd_release squid 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:stretch_mode_enabled false 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:max_osd 12 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6802/4075395622,v1:172.21.15.150:6803/4075395622] [v2:172.21.15.150:6804/4075395622,v1:172.21.15.150:6805/4075395622] exists,up 34d33d5d-faae-42ac-b7d3-70679106ee85 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/737563874,v1:172.21.15.156:6801/737563874] [v2:172.21.15.156:6802/737563874,v1:172.21.15.156:6803/737563874] exists,up 14890f1b-f4e5-4d2d-a1f0-71c84f464fd0 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/1228471627,v1:172.21.15.191:6801/1228471627] [v2:172.21.15.191:6802/1228471627,v1:172.21.15.191:6803/1228471627] exists,up 1f5a563a-5180-40bd-a2a9-625033da15e4 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6810/2684549380,v1:172.21.15.150:6811/2684549380] [v2:172.21.15.150:6812/2684549380,v1:172.21.15.150:6813/2684549380] exists,up 869ada62-adef-48b5-8d36-f57fb0220c2a 2024-03-18T15:59:46.167 INFO:teuthology.orchestra.run.smithi150.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/1341344741,v1:172.21.15.156:6809/1341344741] [v2:172.21.15.156:6810/1341344741,v1:172.21.15.156:6811/1341344741] exists,up 7324a8d6-300d-4163-8282-c368870e2bfd 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/3189109632,v1:172.21.15.191:6809/3189109632] [v2:172.21.15.191:6810/3189109632,v1:172.21.15.191:6811/3189109632] exists,up 53bfde8f-b1e3-4482-ac6a-e17c6a095ea4 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6818/4061900525,v1:172.21.15.150:6819/4061900525] [v2:172.21.15.150:6820/4061900525,v1:172.21.15.150:6821/4061900525] exists,up 7735e727-557c-4104-9d72-5157f53ec694 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/255625842,v1:172.21.15.156:6817/255625842] [v2:172.21.15.156:6818/255625842,v1:172.21.15.156:6819/255625842] exists,up ce254a57-8b1c-4b10-b76e-510130e8d399 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/2388059353,v1:172.21.15.191:6817/2388059353] [v2:172.21.15.191:6818/2388059353,v1:172.21.15.191:6819/2388059353] exists,up aa202baf-1dea-495c-9c8a-637cbbba34c9 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6826/145936256,v1:172.21.15.150:6827/145936256] [v2:172.21.15.150:6828/145936256,v1:172.21.15.150:6829/145936256] exists,up e301a14e-5738-44bf-af14-269a3be744dd 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/4270466180,v1:172.21.15.156:6825/4270466180] [v2:172.21.15.156:6826/4270466180,v1:172.21.15.156:6827/4270466180] exists,up 1f122516-01f0-41e8-9ff8-c73bbf6609d1 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/2841370319,v1:172.21.15.191:6825/2841370319] [v2:172.21.15.191:6826/2841370319,v1:172.21.15.191:6827/2841370319] exists,up 0691d334-c620-436b-b0f8-8343b4c4cf33 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 2.0 [8,1] 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 2.4 [2,11] 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 2.16 [2,4] 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 2.1d [8,11] 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 3.13 [4,9] 2024-03-18T15:59:46.168 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 3.2c [4,9] 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:pg_upmap_items 3.38 [3,9] 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2417146623 expires 2024-03-19T15:50:55.359841+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6801/775411148 expires 2024-03-19T15:50:55.359841+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/716777891 expires 2024-03-19T15:50:55.359841+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6800/775411148 expires 2024-03-19T15:50:55.359841+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6800/2001484430 expires 2024-03-19T15:50:29.895830+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:6801/2001484430 expires 2024-03-19T15:50:29.895830+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/1772653073 expires 2024-03-19T15:50:29.895830+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/3872392632 expires 2024-03-19T15:50:29.895830+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/2882492618 expires 2024-03-19T15:50:29.895830+0000 2024-03-18T15:59:46.169 INFO:teuthology.orchestra.run.smithi150.stdout:blocklist 172.21.15.150:0/520847709 expires 2024-03-19T15:50:55.359841+0000 2024-03-18T15:59:46.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 msgr2=0x7f6a800a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:46.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 0x7f6a800a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a94103380 tx=0x7f6a90009260 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 msgr2=0x7f6a9419fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c007fa0 tx=0x7f6a8c0079a0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 shutdown_connections 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a94103ec0 0x7f6a9419fa70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6a94102320 0x7f6a941984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6a800a2910 0x7f6a800a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.166+0000 7f6a9c638700 1 --2- 172.21.15.150:0/3983389543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a94103520 0x7f6a941989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.167+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 >> 172.21.15.150:0/3983389543 conn(0x7f6a940fd8e0 msgr2=0x7f6a94106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.167+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 shutdown_connections 2024-03-18T15:59:46.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:46.167+0000 7f6a9c638700 1 -- 172.21.15.150:0/3983389543 wait complete. 2024-03-18T15:59:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:46 smithi150 ceph-mon[31946]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:46.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3983389543' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:59:46.931 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T15:59:46.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:46 smithi191 ceph-mon[36784]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:46.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:46 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3983389543' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:59:47.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:46 smithi156 ceph-mon[35919]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:47.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:46 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3983389543' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/1417506328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec1020e0 msgr2=0x7fe3ec108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/1417506328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec108970 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ca0 tx=0x7fe3e402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/1417506328 shutdown_connections 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/1417506328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec108970 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/1417506328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec101720 0x7fe3ec101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/1417506328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec100530 0x7fe3ec100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/1417506328 >> 172.21.15.150:0/1417506328 conn(0x7fe3ec0fbac0 msgr2=0x7fe3ec0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/1417506328 shutdown_connections 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/1417506328 wait complete. 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 Processor -- start 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 -- start start 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.051+0000 7fe3f0d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 0x7fe3ec1967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec10c190 con 0x7fe3ec101720 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec10c010 con 0x7fe3ec100530 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec10c310 con 0x7fe3ec1020e0 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48128/0 (socket says 172.21.15.150:48128) 2024-03-18T15:59:47.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 -- 172.21.15.150:0/2987836209 learned_addr learned my addr 172.21.15.150:0/2987836209 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eb7fe640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 0x7fe3ec1967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3ebfff640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 -- 172.21.15.150:0/2987836209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec1020e0 msgr2=0x7fe3ec19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 -- 172.21.15.150:0/2987836209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 msgr2=0x7fe3ec1967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 0x7fe3ec1967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 -- 172.21.15.150:0/2987836209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec19e400 con 0x7fe3ec101720 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eb7fe640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 0x7fe3ec1967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3eaffd640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000e9f0 tx=0x7fe3e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000cd80 con 0x7fe3ec101720 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e0010910 con 0x7fe3ec101720 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec19e6f0 con 0x7fe3ec101720 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0018990 con 0x7fe3ec101720 2024-03-18T15:59:47.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.052+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec19ec90 con 0x7fe3ec101720 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.053+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec107850 con 0x7fe3ec101720 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.054+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe3e0010ac0 con 0x7fe3ec101720 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.055+0000 7fe3e8ff9640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 0x7fe3d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.055+0000 7fe3eb7fe640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 0x7fe3d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.055+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fe3e00cece0 con 0x7fe3ec101720 2024-03-18T15:59:47.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.055+0000 7fe3eb7fe640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 0x7fe3d40a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec101580 tx=0x7fe3dc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.059+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3e009b760 con 0x7fe3ec101720 2024-03-18T15:59:47.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.237+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3ec0623e0 con 0x7fe3ec101720 2024-03-18T15:59:47.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.238+0000 7fe3e8ff9640 1 -- 172.21.15.150:0/2987836209 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7fe3e00bb020 con 0x7fe3ec101720 2024-03-18T15:59:47.240 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:47.240 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.240+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 msgr2=0x7fe3d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.240+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 0x7fe3d40a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec101580 tx=0x7fe3dc005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.240+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 msgr2=0x7fe3ec196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.240+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000e9f0 tx=0x7fe3e000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 shutdown_connections 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fe3d40a2880 0x7fe3d40a4d40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3ec1020e0 0x7fe3ec19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3ec101720 0x7fe3ec196d20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 --2- 172.21.15.150:0/2987836209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3ec100530 0x7fe3ec1967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 >> 172.21.15.150:0/2987836209 conn(0x7fe3ec0fbac0 msgr2=0x7fe3ec104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 shutdown_connections 2024-03-18T15:59:47.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.241+0000 7fe3f0d9d640 1 -- 172.21.15.150:0/2987836209 wait complete. 2024-03-18T15:59:47.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:59:47.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:59:47.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 -- 172.21.15.150:0/2455336604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 msgr2=0x7fad90101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 --2- 172.21.15.150:0/2455336604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90101d70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fad84009900 tx=0x7fad8402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 -- 172.21.15.150:0/2455336604 shutdown_connections 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 --2- 172.21.15.150:0/2455336604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad901022b0 0x7fad90108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 --2- 172.21.15.150:0/2455336604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90101d70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 --2- 172.21.15.150:0/2455336604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad90100700 0x7fad90100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 -- 172.21.15.150:0/2455336604 >> 172.21.15.150:0/2455336604 conn(0x7fad900fbc90 msgr2=0x7fad900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 -- 172.21.15.150:0/2455336604 shutdown_connections 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.358+0000 7fad962f1640 1 -- 172.21.15.150:0/2455336604 wait complete. 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 Processor -- start 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 -- start start 2024-03-18T15:59:47.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad90100700 0x7fad90196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 0x7fad9019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad9010c360 con 0x7fad901018f0 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad9010c1e0 con 0x7fad90100700 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad962f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fad9010c4e0 con 0x7fad901022b0 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad8f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad8f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48144/0 (socket says 172.21.15.150:48144) 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad8f7fe640 1 -- 172.21.15.150:0/968558433 learned_addr learned my addr 172.21.15.150:0/968558433 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.359+0000 7fad8ffff640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad90100700 0x7fad90196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad9488e640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 0x7fad9019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 -- 172.21.15.150:0/968558433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 msgr2=0x7fad9019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 0x7fad9019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 -- 172.21.15.150:0/968558433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad90100700 msgr2=0x7fad90196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad90100700 0x7fad90196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 -- 172.21.15.150:0/968558433 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9019e630 con 0x7fad901018f0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad9488e640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 0x7fad9019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8f7fe640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fad84009230 tx=0x7fad8402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84041070 con 0x7fad901018f0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9019e8c0 con 0x7fad901018f0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad84004450 con 0x7fad901018f0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84003e20 con 0x7fad901018f0 2024-03-18T15:59:47.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.360+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9019ee00 con 0x7fad901018f0 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.361+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad90107a20 con 0x7fad901018f0 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.362+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fad84044070 con 0x7fad901018f0 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.363+0000 7fad8d7fa640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 0x7fad780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.363+0000 7fad8ffff640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 0x7fad780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.363+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fad840f0690 con 0x7fad901018f0 2024-03-18T15:59:47.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.363+0000 7fad8ffff640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 0x7fad780a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad90101750 tx=0x7fad80005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.367+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad840bd700 con 0x7fad901018f0 2024-03-18T15:59:47.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.540+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fad900623e0 con 0x7fad901018f0 2024-03-18T15:59:47.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.540+0000 7fad8d7fa640 1 -- 172.21.15.150:0/968558433 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fad840bd000 con 0x7fad901018f0 2024-03-18T15:59:47.542 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 msgr2=0x7fad780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 0x7fad780a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad90101750 tx=0x7fad80005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 msgr2=0x7fad90196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fad84009230 tx=0x7fad8402c650 comp rx=0 tx=0).stop 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 shutdown_connections 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fad780a2800 0x7fad780a4cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad901022b0 0x7fad9019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad901018f0 0x7fad90196f50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 --2- 172.21.15.150:0/968558433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad90100700 0x7fad90196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 >> 172.21.15.150:0/968558433 conn(0x7fad900fbc90 msgr2=0x7fad90104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 shutdown_connections 2024-03-18T15:59:47.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.543+0000 7fad962f1640 1 -- 172.21.15.150:0/968558433 wait complete. 2024-03-18T15:59:47.555 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 -- 172.21.15.150:0/3152500181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc102290 msgr2=0x7f4fcc108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/3152500181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc102290 0x7f4fcc108b20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009ca0 tx=0x7f4fc402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 -- 172.21.15.150:0/3152500181 shutdown_connections 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/3152500181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc102290 0x7f4fcc108b20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/3152500181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 0x7f4fcc101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/3152500181 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 -- 172.21.15.150:0/3152500181 >> 172.21.15.150:0/3152500181 conn(0x7f4fcc0fbc70 msgr2=0x7f4fcc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 -- 172.21.15.150:0/3152500181 shutdown_connections 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 -- 172.21.15.150:0/3152500181 wait complete. 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.658+0000 7f4fd20ea640 1 Processor -- start 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 -- start start 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 0x7f4fcc1096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc102290 0x7f4fcc109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10d480 con 0x7f4fcc1006e0 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10d300 con 0x7f4fcc1018d0 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc10d600 con 0x7f4fcc102290 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48154/0 (socket says 172.21.15.150:48154) 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 -- 172.21.15.150:0/1396563878 learned_addr learned my addr 172.21.15.150:0/1396563878 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:47.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcbfff640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc102290 0x7f4fcc109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcaffd640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 0x7f4fcc1096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 -- 172.21.15.150:0/1396563878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc102290 msgr2=0x7f4fcc109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc102290 0x7f4fcc109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 -- 172.21.15.150:0/1396563878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 msgr2=0x7f4fcc1096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 0x7f4fcc1096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 -- 172.21.15.150:0/1396563878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc10a470 con 0x7f4fcc1006e0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fcb7fe640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00e9c0 tx=0x7f4fbc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc00cd80 con 0x7f4fcc1006e0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc1a1bf0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fbc0035f0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.659+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc010690 con 0x7f4fcc1006e0 2024-03-18T15:59:47.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.660+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc1a20f0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.660+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc107a00 con 0x7f4fcc1006e0 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.661+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4fbc0037a0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.662+0000 7f4fc8ff9640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.662+0000 7f4fcaffd640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.662+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4fbc0cee10 con 0x7f4fcc1006e0 2024-03-18T15:59:47.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.662+0000 7f4fcaffd640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc10add0 tx=0x7f4fc0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.666+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fbc09bee0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2987836209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:47.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/968558433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:47.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.841+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4fcc02d3f0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.842+0000 7f4fc8ff9640 1 -- 172.21.15.150:0/1396563878 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f4fbc09b7e0 con 0x7f4fcc1006e0 2024-03-18T15:59:47.843 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 msgr2=0x7f4fb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc10add0 tx=0x7f4fc0005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 msgr2=0x7f4fcc10b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00e9c0 tx=0x7f4fbc00ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 shutdown_connections 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fcc102290 0x7f4fcc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fcc1018d0 0x7f4fcc1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 --2- 172.21.15.150:0/1396563878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fcc1006e0 0x7f4fcc10b060 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 >> 172.21.15.150:0/1396563878 conn(0x7f4fcc0fbc70 msgr2=0x7f4fcc104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 shutdown_connections 2024-03-18T15:59:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.844+0000 7f4fd20ea640 1 -- 172.21.15.150:0/1396563878 wait complete. 2024-03-18T15:59:47.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 -- 172.21.15.150:0/3709255517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8104bf0 msgr2=0x7fa1e8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/3709255517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8104bf0 0x7fa1e8106fe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009900 tx=0x7fa1dc02bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 -- 172.21.15.150:0/3709255517 shutdown_connections 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/3709255517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8107520 0x7fa1e8109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/3709255517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8104bf0 0x7fa1e8106fe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/3709255517 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e81046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 -- 172.21.15.150:0/3709255517 >> 172.21.15.150:0/3709255517 conn(0x7fa1e80fbc90 msgr2=0x7fa1e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.959+0000 7fa1ed28c640 1 -- 172.21.15.150:0/3709255517 shutdown_connections 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 -- 172.21.15.150:0/3709255517 wait complete. 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 Processor -- start 2024-03-18T15:59:47.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 -- start start 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8104bf0 0x7fa1e81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e810d460 con 0x7fa1e8107520 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e810d2e0 con 0x7fa1e81022c0 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1ed28c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e810d5e0 con 0x7fa1e8104bf0 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.960+0000 7fa1e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:60674/0 (socket says 172.21.15.150:60674) 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e6ffd640 1 -- 172.21.15.150:0/4255554471 learned_addr learned my addr 172.21.15.150:0/4255554471 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48160/0 (socket says 172.21.15.150:48160) 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e67fc640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8104bf0 0x7fa1e81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 -- 172.21.15.150:0/4255554471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8104bf0 msgr2=0x7fa1e81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8104bf0 0x7fa1e81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 -- 172.21.15.150:0/4255554471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 msgr2=0x7fa1e8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 -- 172.21.15.150:0/4255554471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e819f780 con 0x7fa1e8107520 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e6ffd640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1e77fe640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000e970 tx=0x7fa1e000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e000ccc0 con 0x7fa1e8107520 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e819fa70 con 0x7fa1e8107520 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e0005020 con 0x7fa1e8107520 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0004a00 con 0x7fa1e8107520 2024-03-18T15:59:47.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.961+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e819ff40 con 0x7fa1e8107520 2024-03-18T15:59:47.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2987836209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:47.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/968558433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:47.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.962+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e80fd780 con 0x7fa1e8107520 2024-03-18T15:59:47.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.963+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa1e00051d0 con 0x7fa1e8107520 2024-03-18T15:59:47.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.964+0000 7fa1cffff640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 0x7fa1d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:47.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.964+0000 7fa1e6ffd640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 0x7fa1d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:47.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.964+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fa1e00ce9a0 con 0x7fa1e8107520 2024-03-18T15:59:47.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.964+0000 7fa1e6ffd640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 0x7fa1d00a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800f7d0 tx=0x7fa1d8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:47.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:47.968+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1e009ba70 con 0x7fa1e8107520 2024-03-18T15:59:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2987836209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/968558433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.143+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1e802d200 con 0x7fa1e8107520 2024-03-18T15:59:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.144+0000 7fa1cffff640 1 -- 172.21.15.150:0/4255554471 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa1e009b370 con 0x7fa1e8107520 2024-03-18T15:59:48.146 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T15:59:48.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 msgr2=0x7fa1d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 0x7fa1d00a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800f7d0 tx=0x7fa1d8009500 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 msgr2=0x7fa1e819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000e970 tx=0x7fa1e000ee30 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 shutdown_connections 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fa1d00a2800 0x7fa1d00a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e8107520 0x7fa1e819f110 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e8104bf0 0x7fa1e81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 --2- 172.21.15.150:0/4255554471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1e81022c0 0x7fa1e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 >> 172.21.15.150:0/4255554471 conn(0x7fa1e80fbc90 msgr2=0x7fa1e8102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 shutdown_connections 2024-03-18T15:59:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.146+0000 7fa1ed28c640 1 -- 172.21.15.150:0/4255554471 wait complete. 2024-03-18T15:59:48.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- 172.21.15.150:0/1549783465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf0101800 msgr2=0x7f6bf0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- 172.21.15.150:0/1549783465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf0101800 0x7f6bf0101c20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009900 tx=0x7f6be802bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- 172.21.15.150:0/1549783465 shutdown_connections 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- 172.21.15.150:0/1549783465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf01033b0 0x7f6bf0109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- 172.21.15.150:0/1549783465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01029f0 0x7f6bf0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- 172.21.15.150:0/1549783465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf0101800 0x7f6bf0101c20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- 172.21.15.150:0/1549783465 >> 172.21.15.150:0/1549783465 conn(0x7f6bf00fcd90 msgr2=0x7f6bf00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- 172.21.15.150:0/1549783465 shutdown_connections 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- 172.21.15.150:0/1549783465 wait complete. 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 Processor -- start 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- start start 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 0x7f6bf0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01033b0 0x7f6bf019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010d480 con 0x7f6bf01029f0 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010d300 con 0x7f6bf01033b0 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.260+0000 7f6bf8362640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010d600 con 0x7f6bf0101800 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf60fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 0x7f6bf0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48170/0 (socket says 172.21.15.150:48170) 2024-03-18T15:59:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf60fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 0x7f6bf0197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:43772/0 (socket says 172.21.15.150:43772) 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 -- 172.21.15.150:0/246150473 learned_addr learned my addr 172.21.15.150:0/246150473 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf68ff640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01033b0 0x7f6bf019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 -- 172.21.15.150:0/246150473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 msgr2=0x7f6bf0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 0x7f6bf0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 -- 172.21.15.150:0/246150473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01033b0 msgr2=0x7f6bf019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01033b0 0x7f6bf019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 -- 172.21.15.150:0/246150473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf019f710 con 0x7f6bf01029f0 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf58fd640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6be000e9f0 tx=0x7f6be000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be000cd80 con 0x7f6bf01029f0 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf019fa00 con 0x7f6bf01029f0 2024-03-18T15:59:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6be0010910 con 0x7f6bf01029f0 2024-03-18T15:59:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be00188f0 con 0x7f6bf01029f0 2024-03-18T15:59:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.261+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf019ffa0 con 0x7f6bf01029f0 2024-03-18T15:59:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.262+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf0108b20 con 0x7f6bf01029f0 2024-03-18T15:59:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.263+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6be0010ac0 con 0x7f6bf01029f0 2024-03-18T15:59:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.264+0000 7f6be77fe640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 0x7f6bdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.264+0000 7f6bf60fe640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 0x7f6bdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.264+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f6be00cedd0 con 0x7f6bf01029f0 2024-03-18T15:59:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.264+0000 7f6bf60fe640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 0x7f6bdc0a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.268+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6be009b850 con 0x7f6bf01029f0 2024-03-18T15:59:48.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.483+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf002d1b0 con 0x7f6bf01029f0 2024-03-18T15:59:48.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.484+0000 7f6be77fe640 1 -- 172.21.15.150:0/246150473 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f6be009f8f0 con 0x7f6bf01029f0 2024-03-18T15:59:48.485 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:48.485 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:59:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:48.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 msgr2=0x7f6bdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 0x7f6bdc0a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 msgr2=0x7f6bf0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6be000e9f0 tx=0x7f6be000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 shutdown_connections 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f6bdc0a2800 0x7f6bdc0a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bf01033b0 0x7f6bf019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf01029f0 0x7f6bf0198030 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 --2- 172.21.15.150:0/246150473 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6bf0101800 0x7f6bf0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 >> 172.21.15.150:0/246150473 conn(0x7f6bf00fcd90 msgr2=0x7f6bf0105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 shutdown_connections 2024-03-18T15:59:48.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.486+0000 7f6bf8362640 1 -- 172.21.15.150:0/246150473 wait complete. 2024-03-18T15:59:48.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:48.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 -- 172.21.15.150:0/3597229040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100100700 msgr2=0x7fc100100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3597229040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100100700 0x7fc100100b20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009900 tx=0x7fc0f402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:48.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 -- 172.21.15.150:0/3597229040 shutdown_connections 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3597229040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001022b0 0x7fc100108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3597229040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1001018f0 0x7fc100101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3597229040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100100700 0x7fc100100b20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.601+0000 7fc106d3b640 1 -- 172.21.15.150:0/3597229040 >> 172.21.15.150:0/3597229040 conn(0x7fc1000fbc70 msgr2=0x7fc1000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- 172.21.15.150:0/3597229040 shutdown_connections 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- 172.21.15.150:0/3597229040 wait complete. 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 Processor -- start 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- start start 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc100100700 0x7fc100197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001018f0 0x7fc100198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d4a0 con 0x7fc1001022b0 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d320 con 0x7fc100100700 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc106d3b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d620 con 0x7fc1001018f0 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc1052d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc1052d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48192/0 (socket says 172.21.15.150:48192) 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc1052d8640 1 -- 172.21.15.150:0/3771214894 learned_addr learned my addr 172.21.15.150:0/3771214894 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc0fffff640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001018f0 0x7fc100198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.602+0000 7fc104ad7640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc100100700 0x7fc100197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 -- 172.21.15.150:0/3771214894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001018f0 msgr2=0x7fc100198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001018f0 0x7fc100198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 -- 172.21.15.150:0/3771214894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc100100700 msgr2=0x7fc100197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc100100700 0x7fc100197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 -- 172.21.15.150:0/3771214894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10019f740 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc1052d8640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800eb10 tx=0x7fc0f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8004b20 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10019fa30 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f8004cc0 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8010760 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.603+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10019ff80 con 0x7fc1001022b0 2024-03-18T15:59:48.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.604+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc100107aa0 con 0x7fc1001022b0 2024-03-18T15:59:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.605+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc0f8005020 con 0x7fc1001022b0 2024-03-18T15:59:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.606+0000 7fc0fdffb640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 0x7fc0f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.606+0000 7fc104ad7640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 0x7fc0f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.606+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc0f80cf110 con 0x7fc1001022b0 2024-03-18T15:59:48.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.606+0000 7fc104ad7640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 0x7fc0f00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4000c00 tx=0x7fc0f4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.610+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0f809c1e0 con 0x7fc1001022b0 2024-03-18T15:59:48.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.786+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc10002d390 con 0x7fc1001022b0 2024-03-18T15:59:48.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.788+0000 7fc0fdffb640 1 -- 172.21.15.150:0/3771214894 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7fc0f809bae0 con 0x7fc1001022b0 2024-03-18T15:59:48.789 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:48.790 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 msgr2=0x7fc0f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 0x7fc0f00a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4000c00 tx=0x7fc0f4005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 msgr2=0x7fc10019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800eb10 tx=0x7fc0f800c2d0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 shutdown_connections 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc0f00a2800 0x7fc0f00a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1001022b0 0x7fc10019f0d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1001018f0 0x7fc100198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 --2- 172.21.15.150:0/3771214894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc100100700 0x7fc100197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 >> 172.21.15.150:0/3771214894 conn(0x7fc1000fbc70 msgr2=0x7fc100104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 shutdown_connections 2024-03-18T15:59:48.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.790+0000 7fc106d3b640 1 -- 172.21.15.150:0/3771214894 wait complete. 2024-03-18T15:59:48.802 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:48 smithi150 ceph-mon[31946]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:48.802 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1396563878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:48.802 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4255554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:48.802 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:48 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/246150473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:48.803 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T15:59:48.804 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 -- 172.21.15.150:0/94259430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c1033b0 msgr2=0x7f022c109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 --2- 172.21.15.150:0/94259430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c1033b0 0x7f022c109c40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0224009ca0 tx=0x7f022402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 -- 172.21.15.150:0/94259430 shutdown_connections 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 --2- 172.21.15.150:0/94259430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c1033b0 0x7f022c109c40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 --2- 172.21.15.150:0/94259430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1029f0 0x7f022c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 --2- 172.21.15.150:0/94259430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c101800 0x7f022c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 -- 172.21.15.150:0/94259430 >> 172.21.15.150:0/94259430 conn(0x7f022c0fcd70 msgr2=0x7f022c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 -- 172.21.15.150:0/94259430 shutdown_connections 2024-03-18T15:59:48.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.906+0000 7f02311e6640 1 -- 172.21.15.150:0/94259430 wait complete. 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 Processor -- start 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 -- start start 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 0x7f022c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d460 con 0x7f022c101800 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d2e0 con 0x7f022c1033b0 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.907+0000 7f02311e6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f022c10d5e0 con 0x7f022c1029f0 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:43798/0 (socket says 172.21.15.150:43798) 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 -- 172.21.15.150:0/2999166373 learned_addr learned my addr 172.21.15.150:0/2999166373 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48210/0 (socket says 172.21.15.150:48210) 2024-03-18T15:59:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022b59e640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 0x7f022c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 -- 172.21.15.150:0/2999166373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 msgr2=0x7f022c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 0x7f022c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 -- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 msgr2=0x7f022c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 -- 172.21.15.150:0/2999166373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022c19f690 con 0x7f022c1029f0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022b59e640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 0x7f022c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022ad9d640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f022a59c640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f022000e9f0 tx=0x7f022000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022000cd80 con 0x7f022c1029f0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0220010910 con 0x7f022c1029f0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022c19f980 con 0x7f022c1029f0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220018990 con 0x7f022c1029f0 2024-03-18T15:59:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.908+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f022c19ff20 con 0x7f022c1029f0 2024-03-18T15:59:48.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.909+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022c108b20 con 0x7f022c1029f0 2024-03-18T15:59:48.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.910+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0220010ac0 con 0x7f022c1029f0 2024-03-18T15:59:48.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.911+0000 7f0213fff640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 0x7f02140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:48.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.911+0000 7f022ad9d640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 0x7f02140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:48.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.911+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f02200cece0 con 0x7f022c1029f0 2024-03-18T15:59:48.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.911+0000 7f022ad9d640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 0x7f02140a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f022c102850 tx=0x7f021c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:48.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:48.915+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f022009b760 con 0x7f022c1029f0 2024-03-18T15:59:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:48 smithi191 ceph-mon[36784]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:48 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1396563878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:48 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4255554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:48 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/246150473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:48 smithi156 ceph-mon[35919]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-18T15:59:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:48 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1396563878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:48 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4255554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:48 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/246150473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.088+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f022c02d1b0 con 0x7f022c1029f0 2024-03-18T15:59:49.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.089+0000 7f0213fff640 1 -- 172.21.15.150:0/2999166373 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f02200bb020 con 0x7f022c1029f0 2024-03-18T15:59:49.090 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 msgr2=0x7f02140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 0x7f02140a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f022c102850 tx=0x7f021c005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 msgr2=0x7f022c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f022000e9f0 tx=0x7f022000eeb0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 shutdown_connections 2024-03-18T15:59:49.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f02140a2880 0x7f02140a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f022c1033b0 0x7f022c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f022c1029f0 0x7f022c197fb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 --2- 172.21.15.150:0/2999166373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f022c101800 0x7f022c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 >> 172.21.15.150:0/2999166373 conn(0x7f022c0fcd70 msgr2=0x7f022c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 shutdown_connections 2024-03-18T15:59:49.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.091+0000 7f02311e6640 1 -- 172.21.15.150:0/2999166373 wait complete. 2024-03-18T15:59:49.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T15:59:49.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3142889225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141033a0 msgr2=0x7fd714109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3142889225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141033a0 0x7fd714109c30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd704009ca0 tx=0x7fd70402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3142889225 shutdown_connections 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3142889225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141033a0 0x7fd714109c30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3142889225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 0x7fd714102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3142889225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141017f0 0x7fd714101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3142889225 >> 172.21.15.150:0/3142889225 conn(0x7fd7140fcd80 msgr2=0x7fd7140ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3142889225 shutdown_connections 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3142889225 wait complete. 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 Processor -- start 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.211+0000 7fd71c3b9640 1 -- start start 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 0x7fd714073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141033a0 0x7fd714074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410d470 con 0x7fd7141017f0 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410d2f0 con 0x7fd7141033a0 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd71410d5f0 con 0x7fd7141029e0 2024-03-18T15:59:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48230/0 (socket says 172.21.15.150:48230) 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 -- 172.21.15.150:0/3080738602 learned_addr learned my addr 172.21.15.150:0/3080738602 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd719954640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 0x7fd714073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a956640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141033a0 0x7fd714074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 -- 172.21.15.150:0/3080738602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 msgr2=0x7fd714073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 0x7fd714073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 -- 172.21.15.150:0/3080738602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141033a0 msgr2=0x7fd714074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141033a0 0x7fd714074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 -- 172.21.15.150:0/3080738602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7140749a0 con 0x7fd7141017f0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71a155640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9c0 tx=0x7fd70c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00cd80 con 0x7fd7141017f0 2024-03-18T15:59:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0035f0 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7141a1bd0 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c010690 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.212+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7141a2120 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.213+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714108b10 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.214+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd70c0037a0 con 0x7fd7141017f0 2024-03-18T15:59:49.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.215+0000 7fd7137fe640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.215+0000 7fd719954640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.215+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fd70c0cee40 con 0x7fd7141017f0 2024-03-18T15:59:49.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.215+0000 7fd719954640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd714075300 tx=0x7fd708005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.219+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd70c09bf10 con 0x7fd7141017f0 2024-03-18T15:59:49.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.396+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd71402d3f0 con 0x7fd7141017f0 2024-03-18T15:59:49.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.396+0000 7fd7137fe640 1 -- 172.21.15.150:0/3080738602 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fd70c09b810 con 0x7fd7141017f0 2024-03-18T15:59:49.397 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T15:59:49.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 msgr2=0x7fd6fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd714075300 tx=0x7fd708005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 msgr2=0x7fd714075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9c0 tx=0x7fd70c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 shutdown_connections 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fd6fc0a2800 0x7fd6fc0a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7141033a0 0x7fd714074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7141029e0 0x7fd714073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 --2- 172.21.15.150:0/3080738602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7141017f0 0x7fd714075590 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 >> 172.21.15.150:0/3080738602 conn(0x7fd7140fcd80 msgr2=0x7fd714105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 shutdown_connections 2024-03-18T15:59:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.398+0000 7fd71c3b9640 1 -- 172.21.15.150:0/3080738602 wait complete. 2024-03-18T15:59:49.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- 172.21.15.150:0/2994961511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 msgr2=0x7fbf6c107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- 172.21.15.150:0/2994961511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c107060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009860 tx=0x7fbf5c02bb70 comp rx=0 tx=0).stop 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- 172.21.15.150:0/2994961511 shutdown_connections 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- 172.21.15.150:0/2994961511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 0x7fbf6c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- 172.21.15.150:0/2994961511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c107060 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- 172.21.15.150:0/2994961511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- 172.21.15.150:0/2994961511 >> 172.21.15.150:0/2994961511 conn(0x7fbf6c0fbc90 msgr2=0x7fbf6c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- 172.21.15.150:0/2994961511 shutdown_connections 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- 172.21.15.150:0/2994961511 wait complete. 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 Processor -- start 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- start start 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c1002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 0x7fbf6c100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10d490 con 0x7fbf6c102340 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10d310 con 0x7fbf6c104c70 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.514+0000 7fbf743a9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c10d610 con 0x7fbf6c1075a0 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48242/0 (socket says 172.21.15.150:48242) 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 -- 172.21.15.150:0/411546561 learned_addr learned my addr 172.21.15.150:0/411546561 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:49.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf71944640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c1002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72946640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 0x7fbf6c100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 -- 172.21.15.150:0/411546561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 msgr2=0x7fbf6c100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 0x7fbf6c100830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 -- 172.21.15.150:0/411546561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 msgr2=0x7fbf6c1002f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c1002f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 -- 172.21.15.150:0/411546561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c1010c0 con 0x7fbf6c102340 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf71944640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c1002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf72145640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400e9c0 tx=0x7fbf6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400cde0 con 0x7fbf6c102340 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c1a1c80 con 0x7fbf6c102340 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c1a21a0 con 0x7fbf6c102340 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf640035f0 con 0x7fbf6c102340 2024-03-18T15:59:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.515+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf64010690 con 0x7fbf6c102340 2024-03-18T15:59:49.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.516+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6c068a20 con 0x7fbf6c102340 2024-03-18T15:59:49.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.517+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbf640037a0 con 0x7fbf6c102340 2024-03-18T15:59:49.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.518+0000 7fbf637fe640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.518+0000 7fbf71944640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.518+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fbf640cedd0 con 0x7fbf6c102340 2024-03-18T15:59:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.518+0000 7fbf71944640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 0x7fbf580a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009830 tx=0x7fbf5c02e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.522+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf6409bea0 con 0x7fbf6c102340 2024-03-18T15:59:49.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.699+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbf6c0623e0 con 0x7fbf6c102340 2024-03-18T15:59:49.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.699+0000 7fbf637fe640 1 -- 172.21.15.150:0/411546561 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fbf64009510 con 0x7fbf6c102340 2024-03-18T15:59:49.700 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T15:59:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 msgr2=0x7fbf580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 0x7fbf580a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009830 tx=0x7fbf5c02e040 comp rx=0 tx=0).stop 2024-03-18T15:59:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 msgr2=0x7fbf6c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400e9c0 tx=0x7fbf6400ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 shutdown_connections 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fbf580a2800 0x7fbf580a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf6c1075a0 0x7fbf6c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf6c104c70 0x7fbf6c1002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 --2- 172.21.15.150:0/411546561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c102340 0x7fbf6c101cb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 >> 172.21.15.150:0/411546561 conn(0x7fbf6c0fbc90 msgr2=0x7fbf6c102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 shutdown_connections 2024-03-18T15:59:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.701+0000 7fbf743a9640 1 -- 172.21.15.150:0/411546561 wait complete. 2024-03-18T15:59:49.712 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:49 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3771214894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.712 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:49 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2999166373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:49.713 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:49 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3080738602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:49.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T15:59:49.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 -- 172.21.15.150:0/1534490440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174100780 msgr2=0x7f4174100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 --2- 172.21.15.150:0/1534490440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174100780 0x7f4174100ba0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4164009900 tx=0x7f416402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 -- 172.21.15.150:0/1534490440 shutdown_connections 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 --2- 172.21.15.150:0/1534490440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 0x7f4174108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 --2- 172.21.15.150:0/1534490440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174101970 0x7f4174101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 --2- 172.21.15.150:0/1534490440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174100780 0x7f4174100ba0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 -- 172.21.15.150:0/1534490440 >> 172.21.15.150:0/1534490440 conn(0x7f41740fbd10 msgr2=0x7f41740fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 -- 172.21.15.150:0/1534490440 shutdown_connections 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.814+0000 7f417a6d1640 1 -- 172.21.15.150:0/1534490440 wait complete. 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 Processor -- start 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 -- start start 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174100780 0x7f417406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 0x7f417419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f417410c360 con 0x7f4174101970 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f417410c1e0 con 0x7f4174100780 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f417a6d1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f417410c4e0 con 0x7f4174102330 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f41737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f41737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48252/0 (socket says 172.21.15.150:48252) 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f41737fe640 1 -- 172.21.15.150:0/2264698335 learned_addr learned my addr 172.21.15.150:0/2264698335 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:49.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f4173fff640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174100780 0x7f417406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.815+0000 7f4178c6e640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 0x7f417419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 -- 172.21.15.150:0/2264698335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 msgr2=0x7f417419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 0x7f417419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 -- 172.21.15.150:0/2264698335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174100780 msgr2=0x7f417406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174100780 0x7f417406a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 -- 172.21.15.150:0/2264698335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417419f780 con 0x7f4174101970 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41737fe640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f416800ea60 tx=0x7f416800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416800cd20 con 0x7f4174101970 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41680035f0 con 0x7f4174101970 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168010690 con 0x7f4174101970 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417419fa70 con 0x7f4174101970 2024-03-18T15:59:49.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.816+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f417419ffc0 con 0x7f4174101970 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.817+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4174107b90 con 0x7f4174101970 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.818+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f41680037a0 con 0x7f4174101970 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.819+0000 7f41717fa640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 0x7f415c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.819+0000 7f4173fff640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 0x7f415c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.819+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f41680cede0 con 0x7f4174101970 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.819+0000 7f4173fff640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 0x7f415c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4164000c00 tx=0x7f4164005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:49.822+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f416809bf40 con 0x7f4174101970 2024-03-18T15:59:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:49 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3771214894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:49 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2999166373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:49 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3080738602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:49 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3771214894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:49 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2999166373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T15:59:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:49 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3080738602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T15:59:50.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.034+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f417402d1b0 con 0x7f4174101970 2024-03-18T15:59:50.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.035+0000 7f41717fa640 1 -- 172.21.15.150:0/2264698335 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11887 (secure 0 0 0) 0x7f416809b840 con 0x7f4174101970 2024-03-18T15:59:50.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:50.037 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T15:58:01.194272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/2169559605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2169559605},{"type":"v1","addr":"172.21.15.191:6835","nonce":2169559605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24481,24485,24496]},"id":1}]} 2024-03-18T15:59:50.037 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 msgr2=0x7f415c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 0x7f415c0a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4164000c00 tx=0x7f4164005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 msgr2=0x7f417419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f416800ea60 tx=0x7f416800ee60 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 shutdown_connections 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f415c0a2800 0x7f415c0a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4174102330 0x7f417419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174101970 0x7f417419c880 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 --2- 172.21.15.150:0/2264698335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4174100780 0x7f417406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 >> 172.21.15.150:0/2264698335 conn(0x7f41740fbd10 msgr2=0x7f4174104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 shutdown_connections 2024-03-18T15:59:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.037+0000 7f417a6d1640 1 -- 172.21.15.150:0/2264698335 wait complete. 2024-03-18T15:59:50.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T15:59:50.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/1459237180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc81022b0 msgr2=0x7f4fc8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/1459237180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc81022b0 0x7f4fc8108b40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009ca0 tx=0x7f4fc002c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/1459237180 shutdown_connections 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/1459237180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc81022b0 0x7f4fc8108b40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/1459237180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/1459237180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc8100700 0x7f4fc8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/1459237180 >> 172.21.15.150:0/1459237180 conn(0x7f4fc80fbc90 msgr2=0x7f4fc80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/1459237180 shutdown_connections 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/1459237180 wait complete. 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 Processor -- start 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.150+0000 7f4fcd9d3640 1 -- start start 2024-03-18T15:59:50.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc81022b0 0x7f4fc819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc810d4a0 con 0x7f4fc8100700 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc810d320 con 0x7f4fc81022b0 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc810d620 con 0x7f4fc81018f0 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48268/0 (socket says 172.21.15.150:48268) 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 -- 172.21.15.150:0/566150652 learned_addr learned my addr 172.21.15.150:0/566150652 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fbe7fc640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc77fe640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc81022b0 0x7f4fc819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 -- 172.21.15.150:0/566150652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 msgr2=0x7f4fc8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 -- 172.21.15.150:0/566150652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc81022b0 msgr2=0x7f4fc819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc81022b0 0x7f4fc819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 -- 172.21.15.150:0/566150652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc819f6b0 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fbe7fc640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fc6ffd640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800e9c0 tx=0x7f4fb800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800cde0 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb8010910 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8018a00 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.151+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc819f9a0 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.152+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc819fef0 con 0x7f4fc8100700 2024-03-18T15:59:50.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.152+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc81079c0 con 0x7f4fc8100700 2024-03-18T15:59:50.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.153+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4fb801f020 con 0x7f4fc8100700 2024-03-18T15:59:50.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.154+0000 7f4fcc9d1640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.154+0000 7f4fbe7fc640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.154+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f4fb80cec10 con 0x7f4fc8100700 2024-03-18T15:59:50.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.154+0000 7f4fbe7fc640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8198fa0 tx=0x7f4fb0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.158+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fb809bce0 con 0x7f4fc8100700 2024-03-18T15:59:50.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.338+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fc80623e0 con 0x7f4fc8100700 2024-03-18T15:59:50.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.339+0000 7f4fcc9d1640 1 -- 172.21.15.150:0/566150652 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22662 (secure 0 0 0) 0x7f4fb809b5e0 con 0x7f4fc8100700 2024-03-18T15:59:50.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T15:59:50.343 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T15:58:56.565844+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T15:59:50.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 msgr2=0x7f4fb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8198fa0 tx=0x7f4fb0005f90 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 msgr2=0x7f4fc8197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800e9c0 tx=0x7f4fb800ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 shutdown_connections 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f4fb40a2800 0x7f4fb40a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc81022b0 0x7f4fc819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fc81018f0 0x7f4fc8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 --2- 172.21.15.150:0/566150652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fc8100700 0x7f4fc8197a90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 >> 172.21.15.150:0/566150652 conn(0x7f4fc80fbc90 msgr2=0x7f4fc8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 shutdown_connections 2024-03-18T15:59:50.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.342+0000 7f4fcd9d3640 1 -- 172.21.15.150:0/566150652 wait complete. 2024-03-18T15:59:50.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T15:57:56.512066+0000', 'modified': '2024-03-18T15:58:01.194272+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24496, 'mds_1': 24485, 'mds_2': 24481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24430': {'gid': 24430, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/2692248421', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 2692248421}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 2692248421}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/1894652569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 1894652569}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 1894652569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/2169559605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 2169559605}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 2169559605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.191:6837/720633938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 720633938}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 720633938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.156:6839/219054917', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 219054917}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 219054917}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/3948564183', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 3948564183}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 3948564183}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24496, 'qdb_cluster': [24481, 24485, 24496]} 2024-03-18T15:59:50.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-18T15:59:50.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T15:59:50.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- 172.21.15.150:0/295762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101d10 msgr2=0x7f8278102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- 172.21.15.150:0/295762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101d10 0x7f8278102130 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8268009900 tx=0x7f826802bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:50.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- 172.21.15.150:0/295762452 shutdown_connections 2024-03-18T15:59:50.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- 172.21.15.150:0/295762452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278101320 0x7f8278108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- 172.21.15.150:0/295762452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278100960 0x7f8278100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- 172.21.15.150:0/295762452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101d10 0x7f8278102130 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- 172.21.15.150:0/295762452 >> 172.21.15.150:0/295762452 conn(0x7f82780fb410 msgr2=0x7f82780fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- 172.21.15.150:0/295762452 shutdown_connections 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- 172.21.15.150:0/295762452 wait complete. 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 Processor -- start 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 -- start start 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278100960 0x7f8278197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.457+0000 7f827e01f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278101d10 0x7f827819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f827e01f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f827810c390 con 0x7f8278101320 2024-03-18T15:59:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f827e01f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f827810c210 con 0x7f8278101d10 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f827e01f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f827810c510 con 0x7f8278100960 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48292/0 (socket says 172.21.15.150:48292) 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f82777fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278100960 0x7f8278197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 -- 172.21.15.150:0/2846365905 learned_addr learned my addr 172.21.15.150:0/2846365905 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8277fff640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278101d10 0x7f827819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 -- 172.21.15.150:0/2846365905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278100960 msgr2=0x7f8278197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278100960 0x7f8278197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 -- 172.21.15.150:0/2846365905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278101d10 msgr2=0x7f827819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278101d10 0x7f827819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 -- 172.21.15.150:0/2846365905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827819f740 con 0x7f8278101320 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8276ffd640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f826c00e9c0 tx=0x7f826c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8274ff9640 1 -- 172.21.15.150:0/2846365905 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00cde0 con 0x7f8278101320 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8274ff9640 1 -- 172.21.15.150:0/2846365905 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c010910 con 0x7f8278101320 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827819fa30 con 0x7f8278101320 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f8274ff9640 1 -- 172.21.15.150:0/2846365905 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c018a00 con 0x7f8278101320 2024-03-18T15:59:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.458+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f827819ffd0 con 0x7f8278101320 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.460+0000 7f8274ff9640 1 -- 172.21.15.150:0/2846365905 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f826c01f020 con 0x7f8278101320 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 Processor -- start 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 -- start start 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e62730 0x5578d7e64bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578d7dc0c80 con 0x5578d7e5cbd0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5578d7d69aa0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8280c05740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5578d7dbcd20 con 0x5578d7e62730 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8277fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8277fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48294/0 (socket says 172.21.15.150:48294) 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8274ff9640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 0x7f82600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8277fff640 1 -- 172.21.15.150:0/2198615648 learned_addr learned my addr 172.21.15.150:0/2198615648 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8276ffd640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e62730 0x5578d7e64bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8277fff640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 0x7f82600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 -- 172.21.15.150:0/2198615648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e62730 msgr2=0x5578d7e64bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e62730 0x5578d7e64bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 -- 172.21.15.150:0/2198615648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 msgr2=0x5578d7e5cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f82777fe640 1 -- 172.21.15.150:0/2198615648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d7d8bff0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8274ff9640 1 -- 172.21.15.150:0/2846365905 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f826c0ceb50 con 0x7f8278101320 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.461+0000 7f8277fff640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.462+0000 7f82777fe640 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e620c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f826802c170 tx=0x7f826802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c708a492f4ba283 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.462+0000 7f8277fff640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 0x7f82600a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5578d7e5de20 tx=0x7f8270004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.462+0000 7f825dffb640 1 -- 172.21.15.150:0/2198615648 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826802cb80 con 0x5578d7e5d5c0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.462+0000 7f825dffb640 1 -- 172.21.15.150:0/2198615648 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8268005040 con 0x5578d7e5d5c0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.462+0000 7f825dffb640 1 -- 172.21.15.150:0/2198615648 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82680049b0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- 172.21.15.150:0/2198615648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 msgr2=0x5578d7e620c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e620c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f826802c170 tx=0x7f826802c650 comp rx=0 tx=0).stop 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- 172.21.15.150:0/2198615648 shutdown_connections 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e62730 0x5578d7e64bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e620c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- 172.21.15.150:0/2198615648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e5cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- 172.21.15.150:0/2198615648 >> 172.21.15.150:0/2198615648 conn(0x5578d7e58140 msgr2=0x5578d7e58540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- 172.21.15.150:0/2198615648 shutdown_connections 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- 172.21.15.150:0/2198615648 wait complete. 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- start start 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 Processor -- start 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5d5c0 0x5578d7e672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e65680 con 0x5578d7e5cbd0 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5578d7d7cef0 con 0x5578d7e62730 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e65800 con 0x5578d7e5d5c0 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8280c05740 10 client.0 _ll_register_callbacks cb 0x5578d7d77dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f82777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f82777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48300/0 (socket says 172.21.15.150:48300) 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f82777fe640 1 -- 172.21.15.150:0/3294301217 learned_addr learned my addr 172.21.15.150:0/3294301217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5d5c0 0x5578d7e672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.463+0000 7f8277fff640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 -- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5d5c0 msgr2=0x5578d7e672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5d5c0 0x5578d7e672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 -- 172.21.15.150:0/3294301217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 msgr2=0x5578d7e6e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 -- 172.21.15.150:0/3294301217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d7d89a30 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f82777fe640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8268009900 tx=0x7f82680042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1820e934d0d9e2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268005600 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f8268004730 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8268005040 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8268004930 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8280c05740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5578d7ad52d0 con 0x5578d7e5cbd0 2024-03-18T15:59:50.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.464+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8268005040 con 0x5578d7e5cbd0 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f826804d030 con 0x5578d7e5cbd0 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 10 client.15246 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 20 client.15246 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7e73440 0x5578d7e748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 4 client.15246 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 --> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d7dce1c0 con 0x5578d7e73440 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7e73440 0x5578d7e748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8256ffd640 10 client.15246 ms_handle_connect on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.465+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7e73440 0x5578d7e748b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b4f6cce29a410a server_cookie=48b08cbc5b0e3380 in_seq=0 out_seq=0 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8256ffd640 1 -- 172.21.15.150:0/3294301217 <== mds.2 v2:172.21.15.191:6836/720633938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578d7dce1c0 con 0x5578d7e73440 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8256ffd640 10 client.15246 handle_command_reply: tid=0 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 client.15246 shutdown 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 msgr2=0x5578d7e66db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8268009900 tx=0x7f82680042c0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 shutdown_connections 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7e73440 0x5578d7e748b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8256ffd640 0 client.15246 ms_handle_reset on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5d5c0 0x5578d7e672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 --2- 172.21.15.150:0/3294301217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5cbd0 0x5578d7e66db0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 >> 172.21.15.150:0/3294301217 conn(0x5578d7e58140 msgr2=0x5578d7e57380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 shutdown_connections 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 1 -- 172.21.15.150:0/3294301217 wait complete. 2024-03-18T15:59:50.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.466+0000 7f8280c05740 20 client.15246 trim_cache size 0 max 16384 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 Processor -- start 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 -- start start 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e65c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e6e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e72bf0 con 0x5578d7e62730 2024-03-18T15:59:50.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e732c0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8280c05740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e73990 con 0x5578d7e5cbd0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8277fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8277fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48306/0 (socket says 172.21.15.150:48306) 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8277fff640 1 -- 172.21.15.150:0/310730672 learned_addr learned my addr 172.21.15.150:0/310730672 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f82777fe640 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e65c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.484+0000 7f8276ffd640 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e6e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 -- 172.21.15.150:0/310730672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 msgr2=0x5578d7e65c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e65c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 -- 172.21.15.150:0/310730672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 msgr2=0x5578d7e6e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e6e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 -- 172.21.15.150:0/310730672 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d7b9f6b0 con 0x5578d7e62730 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8277fff640 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8270003fa0 tx=0x7f8270009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55293d27b8388384 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f825d7fa640 1 -- 172.21.15.150:0/310730672 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270013170 con 0x5578d7e62730 2024-03-18T15:59:50.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f825d7fa640 1 -- 172.21.15.150:0/310730672 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8270010e10 con 0x5578d7e62730 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f825d7fa640 1 -- 172.21.15.150:0/310730672 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270022da0 con 0x5578d7e62730 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 -- 172.21.15.150:0/310730672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 msgr2=0x5578d7e61dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8270003fa0 tx=0x7f8270009be0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 -- 172.21.15.150:0/310730672 shutdown_connections 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e62730 0x5578d7e61dd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e5d5c0 0x5578d7e6e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.485+0000 7f8280c05740 1 --2- 172.21.15.150:0/310730672 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e65c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- 172.21.15.150:0/310730672 >> 172.21.15.150:0/310730672 conn(0x5578d7e58140 msgr2=0x5578d7e5bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- 172.21.15.150:0/310730672 shutdown_connections 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- 172.21.15.150:0/310730672 wait complete. 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- start start 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 Processor -- start 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e69590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e74730 con 0x5578d7e5d5c0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e74060 con 0x5578d7e62730 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5578d7e74e00 con 0x5578d7e5cbd0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8280c05740 10 client.0 _ll_register_callbacks cb 0x7f8270009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8276ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8276ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48322/0 (socket says 172.21.15.150:48322) 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8276ffd640 1 -- 172.21.15.150:0/3671273628 learned_addr learned my addr 172.21.15.150:0/3671273628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f8277fff640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.486+0000 7f82777fe640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e69590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 -- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 msgr2=0x5578d7e69590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e69590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 -- 172.21.15.150:0/3671273628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 msgr2=0x5578d7e6caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 -- 172.21.15.150:0/3671273628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d7e6d0e0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8277fff640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8276ffd640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f826c00eff0 tx=0x7f826c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d319f4fe4de27618 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c0a3ca0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f826c097b70 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8270010e10 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f826c010ac0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 10 client.15252 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5578d7e6e4b0 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8270010e10 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f826c0a3880 con 0x5578d7e5d5c0 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 10 client.15252 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 10 client.15252 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 20 client.15252 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7edd9c0 0x5578d7ed9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 4 client.15252 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.487+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 --> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] -- command(tid 0: {"prefix": "status"}) -- 0x5578d7dc8510 con 0x5578d7edd9c0 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f82777fe640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7edd9c0 0x5578d7ed9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8256ffd640 10 client.15252 ms_handle_connect on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f82777fe640 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7edd9c0 0x5578d7ed9e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af5c1770d0a05abb server_cookie=ad5b5107d6df4353 in_seq=0 out_seq=0 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8256ffd640 1 -- 172.21.15.150:0/3671273628 <== mds.2 v2:172.21.15.191:6836/720633938 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5578d7dc8510 con 0x5578d7edd9c0 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8256ffd640 10 client.15252 handle_command_reply: tid=0 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 client.15252 shutdown 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 msgr2=0x5578d7e69ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f826c00eff0 tx=0x7f826c086040 comp rx=0 tx=0).stop 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 shutdown_connections 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:6836/720633938,v1:172.21.15.191:6837/720633938] conn(0x5578d7edd9c0 0x5578d7ed9e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5578d7e62730 0x5578d7e6caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8256ffd640 0 client.15252 ms_handle_reset on v2:172.21.15.191:6836/720633938 2024-03-18T15:59:50.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5578d7e5d5c0 0x5578d7e69ad0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 --2- 172.21.15.150:0/3671273628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5578d7e5cbd0 0x5578d7e69590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 >> 172.21.15.150:0/3671273628 conn(0x5578d7e58140 msgr2=0x5578d7e5bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 shutdown_connections 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 1 -- 172.21.15.150:0/3671273628 wait complete. 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.488+0000 7f8280c05740 20 client.15252 trim_cache size 0 max 16384 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 2, 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24481, 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:50.497 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 110.29925673, 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 133.853296753 2024-03-18T15:59:50.498 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 msgr2=0x7f82600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 0x7f82600a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5578d7e5de20 tx=0x7f8270004ab0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 msgr2=0x7f8278198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f826c00e9c0 tx=0x7f826c00ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 shutdown_connections 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f82600a2880 0x7f82600a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8278101d10 0x7f827819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8278101320 0x7f8278198060 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 --2- 172.21.15.150:0/2846365905 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278100960 0x7f8278197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 >> 172.21.15.150:0/2846365905 conn(0x7f82780fb410 msgr2=0x7f82780fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 shutdown_connections 2024-03-18T15:59:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.489+0000 7f827e01f640 1 -- 172.21.15.150:0/2846365905 wait complete. 2024-03-18T15:59:50.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.603+0000 7f66bb6dc640 1 -- 172.21.15.150:0/2718473287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 msgr2=0x7f66b4108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.603+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/2718473287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b4108b40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f66ac009ca0 tx=0x7f66ac02c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- 172.21.15.150:0/2718473287 shutdown_connections 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/2718473287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b4108b40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/2718473287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b41018f0 0x7f66b4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/2718473287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b4100700 0x7f66b4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- 172.21.15.150:0/2718473287 >> 172.21.15.150:0/2718473287 conn(0x7f66b40fbc90 msgr2=0x7f66b40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- 172.21.15.150:0/2718473287 shutdown_connections 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- 172.21.15.150:0/2718473287 wait complete. 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 Processor -- start 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- start start 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b4100700 0x7f66b41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b41018f0 0x7f66b4196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f66b410c380 con 0x7f66b41022b0 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66b410c200 con 0x7f66b41018f0 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66bb6dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f66b410c500 con 0x7f66b4100700 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66b9c79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66b9c79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48340/0 (socket says 172.21.15.150:48340) 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66b9c79640 1 -- 172.21.15.150:0/3415717682 learned_addr learned my addr 172.21.15.150:0/3415717682 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.604+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b41018f0 0x7f66b4196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9478640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b4100700 0x7f66b41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 -- 172.21.15.150:0/3415717682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b4100700 msgr2=0x7f66b41969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b4100700 0x7f66b41969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 -- 172.21.15.150:0/3415717682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b41018f0 msgr2=0x7f66b4196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b41018f0 0x7f66b4196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 -- 172.21.15.150:0/3415717682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b419e610 con 0x7f66b41022b0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66b9c79640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f66ac037040 tx=0x7f66ac032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66aa7fc640 1 -- 172.21.15.150:0/3415717682 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac041020 con 0x7f66b41022b0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66aa7fc640 1 -- 172.21.15.150:0/3415717682 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66ac034cd0 con 0x7f66b41022b0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b419e8a0 con 0x7f66b41022b0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66aa7fc640 1 -- 172.21.15.150:0/3415717682 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac039040 con 0x7f66b41022b0 2024-03-18T15:59:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.605+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b419ed90 con 0x7f66b41022b0 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66aa7fc640 1 -- 172.21.15.150:0/3415717682 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f66ac044020 con 0x7f66b41022b0 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 Processor -- start 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 -- start start 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 0x563ab8b306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b412e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ab8a98c80 con 0x563ab8b315b0 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ab8a41aa0 con 0x563ab8b3eef0 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66be3a0740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563ab8a94d20 con 0x563ab8b30240 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66aa7fc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 0x7f66a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48346/0 (socket says 172.21.15.150:48346) 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66b8c77640 1 -- 172.21.15.150:0/1922709130 learned_addr learned my addr 172.21.15.150:0/1922709130 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66b9478640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 0x563ab8b306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.608+0000 7f66b8c77640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9c79640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 0x7f66a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66aa7fc640 1 -- 172.21.15.150:0/3415717682 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f66ac0f08d0 con 0x7f66b41022b0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 -- 172.21.15.150:0/1922709130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 msgr2=0x563ab8b412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 -- 172.21.15.150:0/1922709130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 msgr2=0x563ab8b2fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 -- 172.21.15.150:0/1922709130 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ab8a63ff0 con 0x563ab8b30240 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b8c77640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9c79640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 0x7f66a00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0bd2c0 tx=0x7f66ac08ce50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f66b9478640 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 0x563ab8b306c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009bf0 tx=0x7f66a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bd8a7df8a899bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f669f7fe640 1 -- 172.21.15.150:0/1922709130 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4009da0 con 0x563ab8b30240 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f669f7fe640 1 -- 172.21.15.150:0/1922709130 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4005020 con 0x563ab8b30240 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.609+0000 7f669f7fe640 1 -- 172.21.15.150:0/1922709130 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4004a50 con 0x563ab8b30240 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- 172.21.15.150:0/1922709130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 msgr2=0x563ab8b306c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 0x563ab8b306c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009bf0 tx=0x7f66a4011810 comp rx=0 tx=0).stop 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- 172.21.15.150:0/1922709130 shutdown_connections 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b30240 0x563ab8b306c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 --2- 172.21.15.150:0/1922709130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b315b0 0x563ab8b2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- 172.21.15.150:0/1922709130 >> 172.21.15.150:0/1922709130 conn(0x563ab8b32350 msgr2=0x563ab8b32750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- 172.21.15.150:0/1922709130 shutdown_connections 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- 172.21.15.150:0/1922709130 wait complete. 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 -- start start 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.610+0000 7f66be3a0740 1 Processor -- start 2024-03-18T15:59:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b30240 0x563ab8b43180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 0x563ab8b4a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b41b20 con 0x563ab8b3eef0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ab8a54ef0 con 0x563ab8b30240 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b41ca0 con 0x563ab8b315b0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 10 client.0 _ll_register_callbacks cb 0x563ab8a4fdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 0x563ab8b4a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:43948/0 (socket says 172.21.15.150:43948) 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 -- 172.21.15.150:0/750166472 learned_addr learned my addr 172.21.15.150:0/750166472 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b8c77640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b30240 0x563ab8b43180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 -- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b30240 msgr2=0x563ab8b43180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b30240 0x563ab8b43180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 -- 172.21.15.150:0/750166472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 msgr2=0x563ab8b4a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 0x563ab8b4a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 -- 172.21.15.150:0/750166472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ab8a61a30 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b8c77640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 0x563ab8b4a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66b9478640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a4011d10 tx=0x7f66a4004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f22c7c50c4893a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f669cff9640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4009da0 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a4005020 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f66a4025470 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66a4025670 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 10 client.25090 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.611+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563ab87ad2d0 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4005020 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f66a40196b0 con 0x563ab8b315b0 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 10 client.25090 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 10 client.25090 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 20 client.25090 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8b4fa00 0x563ab8b50e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 4 client.25090 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 --> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ab8aa61c0 con 0x563ab8b4fa00 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66b8c77640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8b4fa00 0x563ab8b50e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f669cff9640 10 client.25090 ms_handle_connect on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.612+0000 7f66b8c77640 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8b4fa00 0x563ab8b50e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83c030496e45a2c9 server_cookie=eb6c8c7c994b28f in_seq=0 out_seq=0 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f669cff9640 1 -- 172.21.15.150:0/750166472 <== mds.1 v2:172.21.15.156:6838/219054917 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ab8aa61c0 con 0x563ab8b4fa00 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f669cff9640 10 client.25090 handle_command_reply: tid=0 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 client.25090 shutdown 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 msgr2=0x563ab8b436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a4011d10 tx=0x7f66a4004ac0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 shutdown_connections 2024-03-18T15:59:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8b4fa00 0x563ab8b50e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b3eef0 0x563ab8b4a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f669cff9640 0 client.25090 ms_handle_reset on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b436c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 --2- 172.21.15.150:0/750166472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b30240 0x563ab8b43180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 >> 172.21.15.150:0/750166472 conn(0x563ab8b32350 msgr2=0x563ab8b2f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 shutdown_connections 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 1 -- 172.21.15.150:0/750166472 wait complete. 2024-03-18T15:59:50.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.613+0000 7f66be3a0740 20 client.25090 trim_cache size 0 max 16384 2024-03-18T15:59:50.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.631+0000 7f66be3a0740 1 Processor -- start 2024-03-18T15:59:50.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.631+0000 7f66be3a0740 1 -- start start 2024-03-18T15:59:50.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b4a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b3b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b4f1b0 con 0x563ab8b30240 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b4f880 con 0x563ab8b3eef0 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66be3a0740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b4ff50 con 0x563ab8b315b0 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48372/0 (socket says 172.21.15.150:48372) 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 -- 172.21.15.150:0/3629397629 learned_addr learned my addr 172.21.15.150:0/3629397629 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b9478640 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b4a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b3b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 -- 172.21.15.150:0/3629397629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 msgr2=0x563ab8b4a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b4a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 -- 172.21.15.150:0/3629397629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 msgr2=0x563ab8b3b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b3b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 -- 172.21.15.150:0/3629397629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ab88776b0 con 0x563ab8b30240 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x563ab8b31140 tx=0x7f66b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3eaf982bf9cee9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f669effd640 1 -- 172.21.15.150:0/3629397629 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b001c040 con 0x563ab8b30240 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f669effd640 1 -- 172.21.15.150:0/3629397629 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b0003b80 con 0x563ab8b30240 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.632+0000 7f669effd640 1 -- 172.21.15.150:0/3629397629 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b00109f0 con 0x563ab8b30240 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- 172.21.15.150:0/3629397629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 msgr2=0x563ab8b3aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x563ab8b31140 tx=0x7f66b0009780 comp rx=0 tx=0).stop 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- 172.21.15.150:0/3629397629 shutdown_connections 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8b3b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b4a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3629397629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b3aed0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- 172.21.15.150:0/3629397629 >> 172.21.15.150:0/3629397629 conn(0x563ab8b32350 msgr2=0x563ab8b4e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- 172.21.15.150:0/3629397629 shutdown_connections 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- 172.21.15.150:0/3629397629 wait complete. 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 -- start start 2024-03-18T15:59:50.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.633+0000 7f66be3a0740 1 Processor -- start 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b45980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8bb3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b50cf0 con 0x563ab8b30240 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b50620 con 0x563ab8b3eef0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563ab8b513c0 con 0x563ab8b315b0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 10 client.0 _ll_register_callbacks cb 0x7f66b0009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48382/0 (socket says 172.21.15.150:48382) 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 -- 172.21.15.150:0/3318286196 learned_addr learned my addr 172.21.15.150:0/3318286196 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b9478640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b45980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8bb3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 -- 172.21.15.150:0/3318286196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 msgr2=0x563ab8b45980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b45980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 -- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 msgr2=0x563ab8bb3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8bb3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 -- 172.21.15.150:0/3318286196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ab8b45ff0 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b9478640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b45980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66b8c77640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009c60 tx=0x7f66b00056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3a28ac95a36a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f669cff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0030040 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f66b0027db0 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66b0003b80 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66b0027340 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 10 client.15270 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.634+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563ab8bb4da0 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b0003b80 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f66b00363d0 con 0x563ab8b30240 2024-03-18T15:59:50.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 10 client.15270 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 10 client.15270 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 20 client.15270 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8bb9640 0x563ab8bb8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 4 client.15270 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 --> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] -- command(tid 0: {"prefix": "status"}) -- 0x563ab8aa0510 con 0x563ab8bb9640 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66b9478640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8bb9640 0x563ab8bb8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f669cff9640 10 client.15270 ms_handle_connect on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66b9478640 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8bb9640 0x563ab8bb8350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4168bfcf82aaa3b server_cookie=8a489bb820ae8493 in_seq=0 out_seq=0 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f669cff9640 1 -- 172.21.15.150:0/3318286196 <== mds.1 v2:172.21.15.156:6838/219054917 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563ab8aa0510 con 0x563ab8bb9640 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f669cff9640 10 client.15270 handle_command_reply: tid=0 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.635+0000 7f66be3a0740 1 client.15270 shutdown 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 msgr2=0x563ab8b45440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009c60 tx=0x7f66b00056d0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 shutdown_connections 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:6838/219054917,v1:172.21.15.156:6839/219054917] conn(0x563ab8bb9640 0x563ab8bb8350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ab8b3eef0 0x563ab8bb3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f669cff9640 0 client.15270 ms_handle_reset on v2:172.21.15.156:6838/219054917 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563ab8b315b0 0x563ab8b45980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 --2- 172.21.15.150:0/3318286196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ab8b30240 0x563ab8b45440 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 >> 172.21.15.150:0/3318286196 conn(0x563ab8b32350 msgr2=0x563ab8b3aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 shutdown_connections 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 1 -- 172.21.15.150:0/3318286196 wait complete. 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66be3a0740 20 client.15270 trim_cache size 0 max 16384 2024-03-18T15:59:50.644 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 1, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24485, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 111.457031265, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 119.485678187 2024-03-18T15:59:50.645 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:50.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 msgr2=0x7f66a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 0x7f66a00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0bd2c0 tx=0x7f66ac08ce50 comp rx=0 tx=0).stop 2024-03-18T15:59:50.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 msgr2=0x7f66b419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f66ac037040 tx=0x7f66ac032f40 comp rx=0 tx=0).stop 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 shutdown_connections 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f66a00a2800 0x7f66a00a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b41022b0 0x7f66b419dfa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66b41018f0 0x7f66b4196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 --2- 172.21.15.150:0/3415717682 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66b4100700 0x7f66b41969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 >> 172.21.15.150:0/3415717682 conn(0x7f66b40fbc90 msgr2=0x7f66b4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 shutdown_connections 2024-03-18T15:59:50.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.636+0000 7f66bb6dc640 1 -- 172.21.15.150:0/3415717682 wait complete. 2024-03-18T15:59:50.647 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:50 smithi150 ceph-mon[31946]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:50.648 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:50 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/411546561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:50.648 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:50 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2264698335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:50.648 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:50 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/566150652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:50.649 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- 172.21.15.150:0/1404977457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 msgr2=0x7f1c98108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- 172.21.15.150:0/1404977457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c98108b20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009900 tx=0x7f1c8802bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- 172.21.15.150:0/1404977457 shutdown_connections 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- 172.21.15.150:0/1404977457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c98108b20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- 172.21.15.150:0/1404977457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981018d0 0x7f1c98101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- 172.21.15.150:0/1404977457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981006e0 0x7f1c98100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- 172.21.15.150:0/1404977457 >> 172.21.15.150:0/1404977457 conn(0x7f1c980fbc70 msgr2=0x7f1c980fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- 172.21.15.150:0/1404977457 shutdown_connections 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- 172.21.15.150:0/1404977457 wait complete. 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 Processor -- start 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- start start 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981006e0 0x7f1c981969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 0x7f1c98196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9810c360 con 0x7f1c98102290 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9810c1e0 con 0x7f1c981018d0 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9f910640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9810c4e0 con 0x7f1c981006e0 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.750+0000 7f1c9dead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 0x7f1c98196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48402/0 (socket says 172.21.15.150:48402) 2024-03-18T15:59:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 -- 172.21.15.150:0/2004726346 learned_addr learned my addr 172.21.15.150:0/2004726346 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981006e0 0x7f1c981969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 -- 172.21.15.150:0/2004726346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981006e0 msgr2=0x7f1c981969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981006e0 0x7f1c981969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 -- 172.21.15.150:0/2004726346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 msgr2=0x7f1c98196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 0x7f1c98196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 -- 172.21.15.150:0/2004726346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9819e5f0 con 0x7f1c98102290 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 0x7f1c98196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9dead640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c88005ab0 tx=0x7f1c8802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c967fc640 1 -- 172.21.15.150:0/2004726346 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88041070 con 0x7f1c98102290 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c967fc640 1 -- 172.21.15.150:0/2004726346 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88004450 con 0x7f1c98102290 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9819e880 con 0x7f1c98102290 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c967fc640 1 -- 172.21.15.150:0/2004726346 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88003e60 con 0x7f1c98102290 2024-03-18T15:59:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.751+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9819ed70 con 0x7f1c98102290 2024-03-18T15:59:50.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.753+0000 7f1c967fc640 1 -- 172.21.15.150:0/2004726346 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1c88044070 con 0x7f1c98102290 2024-03-18T15:59:50.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 Processor -- start 2024-03-18T15:59:50.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 -- start start 2024-03-18T15:59:50.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 0x560b595f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f8940 0x560b595fadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b59554c80 con 0x560b595f2de0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b594fdaa0 con 0x560b595f8940 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1ca25db740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b59550d20 con 0x560b595f37d0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c967fc640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 0x7f1c800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9d6ac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 0x560b595f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48406/0 (socket says 172.21.15.150:48406) 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 -- 172.21.15.150:0/3060624676 learned_addr learned my addr 172.21.15.150:0/3060624676 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9dead640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 0x7f1c800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f8940 0x560b595fadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c967fc640 1 -- 172.21.15.150:0/2004726346 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c880f0650 con 0x7f1c98102290 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 -- 172.21.15.150:0/3060624676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 msgr2=0x560b595f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 0x560b595f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 -- 172.21.15.150:0/3060624676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f8940 msgr2=0x560b595fadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f8940 0x560b595fadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 -- 172.21.15.150:0/3060624676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b5951fff0 con 0x560b595f2de0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 0x560b595f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c013a00 tx=0x7f1c8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1c1eab83d40527 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.754+0000 7f1c9dead640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 0x7f1c800a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1c880bd060 tx=0x7f1c880e7a70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1c7f7fe640 1 -- 172.21.15.150:0/3060624676 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c004250 con 0x560b595f2de0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1c7f7fe640 1 -- 172.21.15.150:0/3060624676 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c011e00 con 0x560b595f2de0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1c7f7fe640 1 -- 172.21.15.150:0/3060624676 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c00f3c0 con 0x560b595f2de0 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 -- 172.21.15.150:0/3060624676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 msgr2=0x560b595f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c013a00 tx=0x7f1c8c009c00 comp rx=0 tx=0).stop 2024-03-18T15:59:50.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 -- 172.21.15.150:0/3060624676 shutdown_connections 2024-03-18T15:59:50.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f8940 0x560b595fadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f37d0 0x560b595f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 --2- 172.21.15.150:0/3060624676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595f3200 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 -- 172.21.15.150:0/3060624676 >> 172.21.15.150:0/3060624676 conn(0x560b595ee350 msgr2=0x560b595ee750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 -- 172.21.15.150:0/3060624676 shutdown_connections 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.755+0000 7f1ca25db740 1 -- 172.21.15.150:0/3060624676 wait complete. 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 -- start start 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 Processor -- start 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b595fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59604600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b595fb8f0 con 0x560b595f2de0 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b59510ef0 con 0x560b595f37d0 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b595fba70 con 0x560b595f8940 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1ca25db740 10 client.0 _ll_register_callbacks cb 0x560b5950bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48408/0 (socket says 172.21.15.150:48408) 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 -- 172.21.15.150:0/344098206 learned_addr learned my addr 172.21.15.150:0/344098206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b595fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59604600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 -- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 msgr2=0x560b59604600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59604600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 -- 172.21.15.150:0/344098206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 msgr2=0x560b595fd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b595fd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.756+0000 7f1c9ceab640 1 -- 172.21.15.150:0/344098206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b5951da30 con 0x560b595f2de0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0139d0 tx=0x7f1c8c011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e53cc02459eb165 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c025070 con 0x560b595f2de0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c8c005460 con 0x560b595f2de0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c011e00 con 0x560b595f2de0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c036050 con 0x560b595f2de0 2024-03-18T15:59:50.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 10 client.15282 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560b592692d0 con 0x560b595f2de0 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c011e00 con 0x560b595f2de0 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1c8c01d020 con 0x560b595f2de0 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 10 client.15282 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 20 client.15282 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59609820 0x560b5960ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 4 client.15282 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.757+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b595621c0 con 0x560b59609820 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.758+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59609820 0x560b5960ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.758+0000 7f1c7cff9640 10 client.15282 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.758+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59609820 0x560b5960ac90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=296bc78aa0443b52 server_cookie=25192f899c9bc399 in_seq=0 out_seq=0 2024-03-18T15:59:50.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.758+0000 7f1c7cff9640 1 -- 172.21.15.150:0/344098206 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b595621c0 con 0x560b59609820 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.758+0000 7f1c7cff9640 10 client.15282 handle_command_reply: tid=0 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 client.15282 shutdown 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 msgr2=0x560b595fd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0139d0 tx=0x7f1c8c011fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 shutdown_connections 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59609820 0x560b5960ac90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59604600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b595fd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1c7cff9640 0 client.15282 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 --2- 172.21.15.150:0/344098206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f2de0 0x560b595fd080 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 >> 172.21.15.150:0/344098206 conn(0x560b595ee350 msgr2=0x560b595eb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 shutdown_connections 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 1 -- 172.21.15.150:0/344098206 wait complete. 2024-03-18T15:59:50.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.759+0000 7f1ca25db740 20 client.15282 trim_cache size 0 max 16384 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 Processor -- start 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 -- start start 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f2de0 0x560b595f8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b59604590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b59608fd0 con 0x560b595f8940 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b596096a0 con 0x560b595f37d0 2024-03-18T15:59:50.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1ca25db740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b59609d70 con 0x560b595f2de0 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48416/0 (socket says 172.21.15.150:48416) 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 -- 172.21.15.150:0/2267222183 learned_addr learned my addr 172.21.15.150:0/2267222183 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b59604590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f2de0 0x560b595f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 -- 172.21.15.150:0/2267222183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f2de0 msgr2=0x560b595f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f2de0 0x560b595f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 -- 172.21.15.150:0/2267222183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 msgr2=0x560b59604590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b59604590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9ceab640 1 -- 172.21.15.150:0/2267222183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b593336b0 con 0x560b595f8940 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.777+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b59604590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c004400 tx=0x7f1c8c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4430ad66a4ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1c7effd640 1 -- 172.21.15.150:0/2267222183 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c025070 con 0x560b595f8940 2024-03-18T15:59:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1c7effd640 1 -- 172.21.15.150:0/2267222183 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c029040 con 0x560b595f8940 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1c7effd640 1 -- 172.21.15.150:0/2267222183 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c028040 con 0x560b595f8940 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 -- 172.21.15.150:0/2267222183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 msgr2=0x560b59608830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c004400 tx=0x7f1c8c005970 comp rx=0 tx=0).stop 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 -- 172.21.15.150:0/2267222183 shutdown_connections 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f8940 0x560b59608830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f37d0 0x560b59604590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2267222183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f2de0 0x560b595f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 -- 172.21.15.150:0/2267222183 >> 172.21.15.150:0/2267222183 conn(0x560b595ee350 msgr2=0x560b595fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 -- 172.21.15.150:0/2267222183 shutdown_connections 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.778+0000 7f1ca25db740 1 -- 172.21.15.150:0/2267222183 wait complete. 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 -- start start 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 Processor -- start 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f2de0 0x560b595ff790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59602ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b5960ab10 con 0x560b595f37d0 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b5960a440 con 0x560b595f2de0 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b5960b1e0 con 0x560b595f8940 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1ca25db740 10 client.0 _ll_register_callbacks cb 0x7f1c90009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48428/0 (socket says 172.21.15.150:48428) 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 -- 172.21.15.150:0/2112600848 learned_addr learned my addr 172.21.15.150:0/2112600848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f2de0 0x560b595ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59602ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 -- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 msgr2=0x560b59602ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59602ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 -- 172.21.15.150:0/2112600848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f2de0 msgr2=0x560b595ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f2de0 0x560b595ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9d6ac640 1 -- 172.21.15.150:0/2112600848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b59603290 con 0x560b595f37d0 2024-03-18T15:59:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.779+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59602ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c9d6ac640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1c98101730 tx=0x7f1c90003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fb71bdbbe57f49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90004240 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1c900043e0 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c029040 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c9002d070 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 10 client.15288 fetch_fsmap learned FSMap version 21 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560b59604670 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c029040 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1c900313d0 con 0x560b595f37d0 2024-03-18T15:59:50.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 10 client.15288 fetch_fsmap finished waiting for FSMap version 21 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 10 client.15288 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 20 client.15288 populate_metadata read hostname 'smithi150' 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59673d00 0x560b59672a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 4 client.15288 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "status"}) -- 0x560b595f86b0 con 0x560b59673d00 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.780+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59673d00 0x560b59672a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1c7cff9640 10 client.15288 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1c9ceab640 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59673d00 0x560b59672a10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92c3b146104a7252 server_cookie=2653fa280d49722d in_seq=0 out_seq=0 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1c7cff9640 1 -- 172.21.15.150:0/2112600848 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560b595f86b0 con 0x560b59673d00 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1c7cff9640 10 client.15288 handle_command_reply: tid=0 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 client.15288 shutdown 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 msgr2=0x560b595ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1c98101730 tx=0x7f1c90003d40 comp rx=0 tx=0).stop 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 shutdown_connections 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b59673d00 0x560b59672a10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b595f8940 0x560b59602ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b595f37d0 0x560b595ffcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 --2- 172.21.15.150:0/2112600848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b595f2de0 0x560b595ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1c7cff9640 0 client.15288 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 >> 172.21.15.150:0/2112600848 conn(0x560b595ee350 msgr2=0x560b595f15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 shutdown_connections 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 1 -- 172.21.15.150:0/2112600848 wait complete. 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.781+0000 7f1ca25db740 20 client.15288 trim_cache size 0 max 16384 2024-03-18T15:59:50.791 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "0cd032e0-e53f-11ee-95c9-87774f69a715", 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 24496, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 113.622739739, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 21, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 88, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 128.325328611 2024-03-18T15:59:50.792 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 msgr2=0x7f1c800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 0x7f1c800a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1c880bd060 tx=0x7f1c880e7a70 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 msgr2=0x7f1c9819df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c88005ab0 tx=0x7f1c8802c6c0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 shutdown_connections 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1c800a2800 0x7f1c800a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c98102290 0x7f1c9819df80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c981018d0 0x7f1c98196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 --2- 172.21.15.150:0/2004726346 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c981006e0 0x7f1c981969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 >> 172.21.15.150:0/2004726346 conn(0x7f1c980fbc70 msgr2=0x7f1c98104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:50.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 shutdown_connections 2024-03-18T15:59:50.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:50.782+0000 7f1c9f910640 1 -- 172.21.15.150:0/2004726346 wait complete. 2024-03-18T15:59:50.795 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T15:59:50.825 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T15:59:50.825 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T15:59:50.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:50.825 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-18T15:59:50.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:50.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-03-18T15:59:50.910 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-03-18T15:59:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:59:50.911 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-18T15:59:50.911 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T15:59:50.912 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T15:59:50.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T15:59:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:50 smithi191 ceph-mon[36784]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:50 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/411546561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:50 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2264698335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:50.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:50 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/566150652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:51.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:50 smithi156 ceph-mon[35919]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T15:59:51.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:50 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/411546561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T15:59:51.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:50 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2264698335' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T15:59:51.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:50 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/566150652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T15:59:51.190 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:52.690 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:52 smithi150 ceph-mon[31946]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:52.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:52 smithi191 ceph-mon[36784]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:53.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:52 smithi156 ceph-mon[35919]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-18T15:59:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- 172.21.15.150:0/1462234956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41072b0 msgr2=0x7f7ec406aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 --2- 172.21.15.150:0/1462234956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41072b0 0x7f7ec406aa10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009990 tx=0x7f7eb4009ca0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- 172.21.15.150:0/1462234956 shutdown_connections 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 --2- 172.21.15.150:0/1462234956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec4104d40 0x7f7ec41051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 --2- 172.21.15.150:0/1462234956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41072b0 0x7f7ec406aa10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 --2- 172.21.15.150:0/1462234956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec406af50 0x7f7ec406b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- 172.21.15.150:0/1462234956 >> 172.21.15.150:0/1462234956 conn(0x7f7ec4078030 msgr2=0x7f7ec4078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- 172.21.15.150:0/1462234956 shutdown_connections 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- 172.21.15.150:0/1462234956 wait complete. 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 Processor -- start 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.072+0000 7f7eca749700 1 -- start start 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 0x7f7ec419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec41072b0 0x7f7ec41a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4118bc0 con 0x7f7ec4104d40 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4118a40 con 0x7f7ec41072b0 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4118d40 con 0x7f7ec406af50 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44374/0 (socket says 172.21.15.150:44374) 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 -- 172.21.15.150:0/3215141683 learned_addr learned my addr 172.21.15.150:0/3215141683 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec3fff700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 0x7f7ec419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec8ce6700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec41072b0 0x7f7ec41a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 -- 172.21.15.150:0/3215141683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 msgr2=0x7f7ec419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 0x7f7ec419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 -- 172.21.15.150:0/3215141683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec41072b0 msgr2=0x7f7ec41a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec41072b0 0x7f7ec41a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 -- 172.21.15.150:0/3215141683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec41a3f60 con 0x7f7ec4104d40 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec3fff700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 0x7f7ec419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec37fe700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800e9a0 tx=0x7f7eb800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800ca10 con 0x7f7ec4104d40 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb800cbb0 con 0x7f7ec4104d40 2024-03-18T15:59:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec41a4250 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8018680 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.073+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec41abad0 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.075+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec4101880 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.076+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7eb8010430 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.076+0000 7f7ec17fa700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 0x7f7eac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.076+0000 7f7ec3fff700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 0x7f7eac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.076+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f7eb80cec70 con 0x7f7ec4104d40 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.077+0000 7f7ec3fff700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 0x7f7eac0a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400b2e0 tx=0x7f7eb4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.081+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7eb809bd40 con 0x7f7ec4104d40 2024-03-18T15:59:53.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.321+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7ec41abca0 con 0x7f7ec4104d40 2024-03-18T15:59:53.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.321+0000 7f7ec17fa700 1 -- 172.21.15.150:0/3215141683 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7f7eb8097030 con 0x7f7ec4104d40 2024-03-18T15:59:53.323 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 msgr2=0x7f7eac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 0x7f7eac0a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400b2e0 tx=0x7f7eb4005960 comp rx=0 tx=0).stop 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 msgr2=0x7f7ec419f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800e9a0 tx=0x7f7eb800ecb0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 shutdown_connections 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f7eac0a2890 0x7f7eac0a4d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ec41072b0 0x7f7ec41a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7ec406af50 0x7f7ec419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 --2- 172.21.15.150:0/3215141683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec4104d40 0x7f7ec419f490 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 >> 172.21.15.150:0/3215141683 conn(0x7f7ec4078030 msgr2=0x7f7ec4103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 shutdown_connections 2024-03-18T15:59:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:53.324+0000 7f7eca749700 1 -- 172.21.15.150:0/3215141683 wait complete. 2024-03-18T15:59:53.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:53 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3215141683' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:59:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:53 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3215141683' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:59:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:53 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3215141683' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T15:59:54.130 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T15:59:54.130 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T15:59:54.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T15:59:54.131 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T15:59:54.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:54.132 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-18T15:59:54.148 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T15:59:54.148 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T15:59:54.148 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-18T15:59:54.149 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T15:59:54.150 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42128sec preferred_lft 42128sec 2024-03-18T15:59:54.150 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-18T15:59:54.150 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-18T15:59:54.150 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T15:59:54.150 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:59:54.150 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-18T15:59:54.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-03-18T15:59:54.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-03-18T15:59:54.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-03-18T15:59:54.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T15:59:54.151 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-18T15:59:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:54.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:54.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:54.358 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T15:59:54.385 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-18T15:59:54.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:54.387 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-18T15:59:54.462 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-18T15:59:54.462 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T15:59:54.463 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T15:59:54.463 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T15:59:54.463 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T15:59:54.463 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:59:54.463 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-18T15:59:54.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T15:59:54.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T15:59:54.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T15:59:54.464 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-18T15:59:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:54.605 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:54 smithi150 ceph-mon[31946]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:54.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:54.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:54.762 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-18T15:59:54.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:54.778 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-03-18T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-18T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T15:59:54.833 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-18T15:59:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:54.956 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T15:59:54.957 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-18T15:59:54.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:54 smithi191 ceph-mon[36784]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:55.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:55.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:54 smithi156 ceph-mon[35919]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-18T15:59:55.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.104 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T15:59:55.104 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-18T15:59:55.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-18T15:59:55.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T15:59:55.104 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-18T15:59:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:55.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.186 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:59:55.186 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T15:59:55.200 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T15:59:55.201 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T15:59:55.201 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T15:59:55.253 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T15:59:55.253 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T15:59:55.253 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/3b1f2cfb-cb65-41f3-9829-04ee269f3064 2024-03-18T15:59:55.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:55.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/3b1f2cfb-cb65-41f3-9829-04ee269f3064 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T15:59:55.339 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T15:59:55.339 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-18T15:59:55.339 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.150:6789,172.21.15.156:6789,172.21.15.191:6789" 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.150:6789,172.21.15.156:6789,172.21.15.191:6789:/volumes/qa/sv_1/3b1f2cfb-cb65-41f3-9829-04ee269f3064 2024-03-18T15:59:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=0cd032e0-e53f-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T15:59:55.450 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T15:59:55.450 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-03-18T15:59:55.450 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T15:59:55.450 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T15:59:55.450 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T15:59:55.452 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T15:59:55.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T15:59:55.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T15:59:55.485 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:59:55.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:59:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.583 INFO:teuthology.orchestra.run.smithi150.stdout:client25123 2024-03-18T15:59:55.583 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T15:59:55.583 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T15:59:55.600 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:59:55.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:59:55.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.696 INFO:teuthology.orchestra.run.smithi150.stdout:client25123 2024-03-18T15:59:55.696 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T15:59:55.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T15:59:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.752 INFO:teuthology.orchestra.run.smithi150.stdout:client25123 2024-03-18T15:59:55.752 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715.client25123/status') 2024-03-18T15:59:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T15:59:55.813 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.25123 (3)172.21.15.150:0/1634258335 2024-03-18T15:59:55.814 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2024-03-18T15:59:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2024-03-18T15:59:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2024-03-18T15:59:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:72 bytes copied, 2.4252e-05 s, 3.0 MB/s 2024-03-18T15:59:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T15:59:55.821 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T15:59:55.834 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:59:55.835 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T15:59:55.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 -- 172.21.15.150:0/2888152032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 msgr2=0x7f0a1c073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 --2- 172.21.15.150:0/2888152032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c073bf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009900 tx=0x7f0a0c02bcd0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 -- 172.21.15.150:0/2888152032 shutdown_connections 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 --2- 172.21.15.150:0/2888152032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 0x7f0a1c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 --2- 172.21.15.150:0/2888152032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 0x7f0a1c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 --2- 172.21.15.150:0/2888152032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c073bf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 -- 172.21.15.150:0/2888152032 >> 172.21.15.150:0/2888152032 conn(0x7f0a1c0fcda0 msgr2=0x7f0a1c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.936+0000 7f0a22085640 1 -- 172.21.15.150:0/2888152032 shutdown_connections 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 -- 172.21.15.150:0/2888152032 wait complete. 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 Processor -- start 2024-03-18T15:59:55.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 -- start start 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 0x7f0a1c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 0x7f0a1c1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10f690 con 0x7f0a1c0757a0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10f510 con 0x7f0a1c1096c0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a22085640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c10f810 con 0x7f0a1c0741c0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a1affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a1affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44384/0 (socket says 172.21.15.150:44384) 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.937+0000 7f0a1affd640 1 -- 172.21.15.150:0/1539061258 learned_addr learned my addr 172.21.15.150:0/1539061258 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1b7fe640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 0x7f0a1c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1bfff640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 0x7f0a1c1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 -- 172.21.15.150:0/1539061258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 msgr2=0x7f0a1c199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 0x7f0a1c199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 -- 172.21.15.150:0/1539061258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 msgr2=0x7f0a1c1a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 0x7f0a1c1a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 -- 172.21.15.150:0/1539061258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c1a18e0 con 0x7f0a1c0757a0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a1affd640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000e9f0 tx=0x7f0a1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1000ccc0 con 0x7f0a1c0757a0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1c1a1bd0 con 0x7f0a1c0757a0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1000ce60 con 0x7f0a1c0757a0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10010690 con 0x7f0a1c0757a0 2024-03-18T15:59:55.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.938+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1c1a2120 con 0x7f0a1c0757a0 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.939+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a1c108480 con 0x7f0a1c0757a0 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.940+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0a100046c0 con 0x7f0a1c0757a0 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.941+0000 7f0a18ff9640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 0x7f0a040a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.941+0000 7f0a1b7fe640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 0x7f0a040a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.941+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f0a100cede0 con 0x7f0a1c0757a0 2024-03-18T15:59:55.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.941+0000 7f0a1b7fe640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 0x7f0a040a4d60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c19ab70 tx=0x7f0a0c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:55.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:55.945+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a1009beb0 con 0x7f0a1c0757a0 2024-03-18T15:59:56.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.159+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f0a1c0623e0 con 0x7f0a1c0757a0 2024-03-18T15:59:56.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.577+0000 7f0a18ff9640 1 -- 172.21.15.150:0/1539061258 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f0a1009b7b0 con 0x7f0a1c0757a0 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.579+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 msgr2=0x7f0a040a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.579+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 0x7f0a040a4d60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c19ab70 tx=0x7f0a0c000bc0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.579+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 msgr2=0x7f0a1c19a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.579+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000e9f0 tx=0x7f0a1000edf0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 shutdown_connections 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f0a040a28a0 0x7f0a040a4d60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a1c1096c0 0x7f0a1c1a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a1c0757a0 0x7f0a1c19a1e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 --2- 172.21.15.150:0/1539061258 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0741c0 0x7f0a1c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 >> 172.21.15.150:0/1539061258 conn(0x7f0a1c0fcda0 msgr2=0x7f0a1c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:56.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 shutdown_connections 2024-03-18T15:59:56.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.580+0000 7f0a22085640 1 -- 172.21.15.150:0/1539061258 wait complete. 2024-03-18T15:59:56.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-18T15:59:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- 172.21.15.150:0/4239287787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6341018f0 msgr2=0x7fc634101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 --2- 172.21.15.150:0/4239287787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6341018f0 0x7fc634101d70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc628009900 tx=0x7fc62802bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- 172.21.15.150:0/4239287787 shutdown_connections 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 --2- 172.21.15.150:0/4239287787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341022b0 0x7fc634108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 --2- 172.21.15.150:0/4239287787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6341018f0 0x7fc634101d70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 --2- 172.21.15.150:0/4239287787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc634100700 0x7fc634100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- 172.21.15.150:0/4239287787 >> 172.21.15.150:0/4239287787 conn(0x7fc6340fbc70 msgr2=0x7fc6340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- 172.21.15.150:0/4239287787 shutdown_connections 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- 172.21.15.150:0/4239287787 wait complete. 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 Processor -- start 2024-03-18T15:59:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.699+0000 7fc639b90640 1 -- start start 2024-03-18T15:59:56.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:56.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341018f0 0x7fc634197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6341022b0 0x7fc63419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc63410c380 con 0x7fc634100700 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc63410c200 con 0x7fc6341022b0 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc63410c500 con 0x7fc6341018f0 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44402/0 (socket says 172.21.15.150:44402) 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 -- 172.21.15.150:0/2423997552 learned_addr learned my addr 172.21.15.150:0/2423997552 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc632ffd640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341018f0 0x7fc634197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc633fff640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6341022b0 0x7fc63419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 -- 172.21.15.150:0/2423997552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341018f0 msgr2=0x7fc634197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341018f0 0x7fc634197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 -- 172.21.15.150:0/2423997552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6341022b0 msgr2=0x7fc63419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6341022b0 0x7fc63419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 -- 172.21.15.150:0/2423997552 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63419f6c0 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc6337fe640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e9c0 tx=0x7fc62400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62400cde0 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc624010910 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63419f9b0 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624018a00 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.700+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63419ff20 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.701+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6341a0150 con 0x7fc634100700 2024-03-18T15:59:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.702+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc62401f020 con 0x7fc634100700 2024-03-18T15:59:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.703+0000 7fc630ff9640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 0x7fc61c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.703+0000 7fc632ffd640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 0x7fc61c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.703+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fc6240ceb90 con 0x7fc634100700 2024-03-18T15:59:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.703+0000 7fc632ffd640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 0x7fc61c0a4de0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc62802c7b0 tx=0x7fc628005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:56.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.707+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc62409bc60 con 0x7fc634100700 2024-03-18T15:59:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-mon[31946]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:56.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:59:56.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.911+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc634069570 con 0x7fc61c0a2920 2024-03-18T15:59:56.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.924+0000 7fc630ff9640 1 -- 172.21.15.150:0/2423997552 <== mgr.14150 v2:172.21.15.150:6800/3095497318 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc634069570 con 0x7fc61c0a2920 2024-03-18T15:59:56.926 INFO:teuthology.orchestra.run.smithi150.stdout:{} 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 msgr2=0x7fc61c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 0x7fc61c0a4de0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc62802c7b0 tx=0x7fc628005fd0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 msgr2=0x7fc634197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e9c0 tx=0x7fc62400ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 shutdown_connections 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fc61c0a2920 0x7fc61c0a4de0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6341022b0 0x7fc63419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6341018f0 0x7fc634197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 --2- 172.21.15.150:0/2423997552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc634100700 0x7fc634197aa0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.926+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 >> 172.21.15.150:0/2423997552 conn(0x7fc6340fbc70 msgr2=0x7fc634068e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.927+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 shutdown_connections 2024-03-18T15:59:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:56.927+0000 7fc639b90640 1 -- 172.21.15.150:0/2423997552 wait complete. 2024-03-18T15:59:56.940 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T15:59:56.951 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T15:59:57.037 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T15:59:57.037 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T15:59:57.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-18T15:59:57.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:56 smithi156 ceph-mon[35919]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:57.038 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:56 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:59:57.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:56 smithi191 ceph-mon[36784]: pgmap v348: 97 pgs: 97 active+clean; 639 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-18T15:59:57.038 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:56 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 -- 172.21.15.150:0/2424109063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 msgr2=0x7f7200102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 --2- 172.21.15.150:0/2424109063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200102f50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f71f0009930 tx=0x7f71f002bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 -- 172.21.15.150:0/2424109063 shutdown_connections 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 --2- 172.21.15.150:0/2424109063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7200103490 0x7f7200109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 --2- 172.21.15.150:0/2424109063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200102f50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 --2- 172.21.15.150:0/2424109063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72000ffb00 0x7f72000fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 -- 172.21.15.150:0/2424109063 >> 172.21.15.150:0/2424109063 conn(0x7f72000fb430 msgr2=0x7f72000fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 -- 172.21.15.150:0/2424109063 shutdown_connections 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.139+0000 7f72084e9640 1 -- 172.21.15.150:0/2424109063 wait complete. 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 Processor -- start 2024-03-18T15:59:57.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 -- start start 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7200103490 0x7f720019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f720010d4c0 con 0x7f7200102ad0 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f720010d340 con 0x7f7200103490 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f72084e9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f720010d640 con 0x7f72000ffb00 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7205a84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:42628/0 (socket says 172.21.15.150:42628) 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 -- 172.21.15.150:0/633030614 learned_addr learned my addr 172.21.15.150:0/633030614 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206a86640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7200103490 0x7f720019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 -- 172.21.15.150:0/633030614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7200103490 msgr2=0x7f720019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7200103490 0x7f720019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 -- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 msgr2=0x7f7200198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 -- 172.21.15.150:0/633030614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720019f6d0 con 0x7f72000ffb00 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7205a84640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.140+0000 7f7206285640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9c0 tx=0x7f71fc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.141+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00cd80 con 0x7f72000ffb00 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.141+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720019f9c0 con 0x7f72000ffb00 2024-03-18T15:59:57.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.141+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71fc010910 con 0x7f72000ffb00 2024-03-18T15:59:57.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.141+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720019ff30 con 0x7f72000ffb00 2024-03-18T15:59:57.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.141+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0049c0 con 0x7f72000ffb00 2024-03-18T15:59:57.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.142+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7200108c00 con 0x7f72000ffb00 2024-03-18T15:59:57.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.143+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f71fc01f020 con 0x7f72000ffb00 2024-03-18T15:59:57.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.143+0000 7f71f77fe640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 0x7f71ec0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.144+0000 7f7205a84640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 0x7f71ec0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.144+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f71fc0cf290 con 0x7f72000ffb00 2024-03-18T15:59:57.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.144+0000 7f7205a84640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 0x7f71ec0a4d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71f002efd0 tx=0x7f71f0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:57.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.151+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f71fc026030 con 0x7f72000ffb00 2024-03-18T15:59:57.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:56.915+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:59:57.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:56.915+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:59:57.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:56.915+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:59:57.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:56.915+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:59:57.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:56.915+0000 7fe66b648700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T15:59:57.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.380+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f72000623e0 con 0x7f72000ffb00 2024-03-18T15:59:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.596+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mgrmap(e 19) ==== 137308+0+0 (secure 0 0 0) 0x7f71fc010430 con 0x7f72000ffb00 2024-03-18T15:59:57.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.598+0000 7f71f77fe640 1 -- 172.21.15.150:0/633030614 <== mon.1 v2:172.21.15.191:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f71fc09c360 con 0x7f72000ffb00 2024-03-18T15:59:57.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 msgr2=0x7f71ec0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 0x7f71ec0a4d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71f002efd0 tx=0x7f71f0000bc0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 msgr2=0x7f7200197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9c0 tx=0x7f71fc00edc0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 shutdown_connections 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f71ec0a28a0 0x7f71ec0a4d60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7200103490 0x7f720019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7200102ad0 0x7f7200198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 --2- 172.21.15.150:0/633030614 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72000ffb00 0x7f7200197b00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 >> 172.21.15.150:0/633030614 conn(0x7f72000fb430 msgr2=0x7f7200105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 shutdown_connections 2024-03-18T15:59:57.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.600+0000 7f72084e9640 1 -- 172.21.15.150:0/633030614 wait complete. 2024-03-18T15:59:57.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-18T15:59:57.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 -- 172.21.15.150:0/2161259294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941022b0 msgr2=0x7ff394108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 --2- 172.21.15.150:0/2161259294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941022b0 0x7ff394108b40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff384009ca0 tx=0x7ff38402c0f0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 -- 172.21.15.150:0/2161259294 shutdown_connections 2024-03-18T15:59:57.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 --2- 172.21.15.150:0/2161259294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941022b0 0x7ff394108b40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 --2- 172.21.15.150:0/2161259294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941018f0 0x7ff394101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 --2- 172.21.15.150:0/2161259294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 -- 172.21.15.150:0/2161259294 >> 172.21.15.150:0/2161259294 conn(0x7ff3940fbc70 msgr2=0x7ff3940fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 -- 172.21.15.150:0/2161259294 shutdown_connections 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.721+0000 7ff3991d3640 1 -- 172.21.15.150:0/2161259294 wait complete. 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 Processor -- start 2024-03-18T15:59:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 -- start start 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 0x7ff39419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff39410c380 con 0x7ff3941018f0 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff39410c200 con 0x7ff394100700 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.722+0000 7ff3991d3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff39410c500 con 0x7ff3941022b0 2024-03-18T15:59:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44470/0 (socket says 172.21.15.150:44470) 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39259c640 1 -- 172.21.15.150:0/3508616392 learned_addr learned my addr 172.21.15.150:0/3508616392 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 0x7ff39419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff392d9d640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 -- 172.21.15.150:0/3508616392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 msgr2=0x7ff394196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 -- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 msgr2=0x7ff394196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 -- 172.21.15.150:0/3508616392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39419e550 con 0x7ff3941022b0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39259c640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff392d9d640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff39359e640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 0x7ff39419dee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff38402c630 tx=0x7ff384032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384041020 con 0x7ff3941022b0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39419e7e0 con 0x7ff3941022b0 2024-03-18T15:59:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384034cd0 con 0x7ff3941022b0 2024-03-18T15:59:57.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.723+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384039040 con 0x7ff3941022b0 2024-03-18T15:59:57.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.724+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff39419ecf0 con 0x7ff3941022b0 2024-03-18T15:59:57.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.724+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3941079c0 con 0x7ff3941022b0 2024-03-18T15:59:57.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.726+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 19) ==== 137308+0+0 (secure 0 0 0) 0x7ff384044020 con 0x7ff3941022b0 2024-03-18T15:59:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.726+0000 7ff37bfff640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 0x7ff37c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.726+0000 7ff392d9d640 1 -- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 msgr2=0x7ff37c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.726+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7ff3840f0cd0 con 0x7ff3941022b0 2024-03-18T15:59:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.726+0000 7ff392d9d640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 0x7ff37c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:59:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.730+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff38404b020 con 0x7ff3941022b0 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setuser ceph since I am not root 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: ignoring --setgroup ceph since I am not root 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/633030614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:57.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:57 smithi150 ceph-mon[31946]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:57.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.915+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff3940623e0 con 0x7ff3941022b0 2024-03-18T15:59:57.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.921+0000 7ff37bfff640 1 -- 172.21.15.150:0/3508616392 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7ff3840bddd0 con 0x7ff3941022b0 2024-03-18T15:59:57.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 msgr2=0x7ff37c0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:59:57.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 0x7ff37c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 msgr2=0x7ff39419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 0x7ff39419dee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff38402c630 tx=0x7ff384032f20 comp rx=0 tx=0).stop 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 shutdown_connections 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7ff37c0a28f0 0x7ff37c0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3941022b0 0x7ff39419dee0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3941018f0 0x7ff394196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 --2- 172.21.15.150:0/3508616392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394100700 0x7ff394196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 >> 172.21.15.150:0/3508616392 conn(0x7ff3940fbc70 msgr2=0x7ff394104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 shutdown_connections 2024-03-18T15:59:57.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:57.923+0000 7ff3991d3640 1 -- 172.21.15.150:0/3508616392 wait complete. 2024-03-18T15:59:57.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-18T15:59:57.962 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: ignoring --setuser ceph since I am not root 2024-03-18T15:59:57.962 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: ignoring --setgroup ceph since I am not root 2024-03-18T15:59:57.962 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-mgr[38642]: -- 172.21.15.191:0/566790740 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556116b5e5a0 con 0x556116b32800 2024-03-18T15:59:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:59:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:59:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/633030614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:57 smithi191 ceph-mon[36784]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1539061258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-18T15:59:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T15:59:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/633030614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-mon[35919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-18T15:59:58.018 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: ignoring --setuser ceph since I am not root 2024-03-18T15:59:58.019 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: ignoring --setgroup ceph since I am not root 2024-03-18T15:59:58.019 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:57 smithi156 ceph-mgr[37305]: -- 172.21.15.156:0/1044463844 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fef498e5a0 con 0x55fef4962c00 2024-03-18T15:59:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4180913054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74100750 msgr2=0x7f1d74100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4180913054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74100750 0x7f1d74100b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009860 tx=0x7f1d6802bb70 comp rx=0 tx=0).stop 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4180913054 shutdown_connections 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4180913054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74102300 0x7f1d74108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4180913054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d74101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4180913054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74100750 0x7f1d74100b70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4180913054 >> 172.21.15.150:0/4180913054 conn(0x7f1d740fbcc0 msgr2=0x7f1d740fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4180913054 shutdown_connections 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.040+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4180913054 wait complete. 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 Processor -- start 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 -- start start 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 0x7f1d7406ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 0x7f1d7419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7410c3d0 con 0x7f1d74101940 2024-03-18T15:59:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7410c250 con 0x7f1d74102300 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d7aeb5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7410c550 con 0x7f1d74100750 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d78c51640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 0x7f1d7406ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44482/0 (socket says 172.21.15.150:44482) 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d79452640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 0x7f1d7419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 -- 172.21.15.150:0/4086683980 learned_addr learned my addr 172.21.15.150:0/4086683980 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 -- 172.21.15.150:0/4086683980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 msgr2=0x7f1d7406ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 0x7f1d7406ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 -- 172.21.15.150:0/4086683980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 msgr2=0x7f1d7419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 0x7f1d7419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d73fff640 1 -- 172.21.15.150:0/4086683980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7419f740 con 0x7f1d74101940 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d79452640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 0x7f1d7419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.041+0000 7f1d78c51640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 0x7f1d7406ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T15:59:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d73fff640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000e9c0 tx=0x7f1d6000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6000cde0 con 0x7f1d74101940 2024-03-18T15:59:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d600035f0 con 0x7f1d74101940 2024-03-18T15:59:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7419f9d0 con 0x7f1d74101940 2024-03-18T15:59:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d60010690 con 0x7f1d74101940 2024-03-18T15:59:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.042+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7419ff20 con 0x7f1d74101940 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.043+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d74107a70 con 0x7f1d74101940 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.044+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 137308+0+0 (secure 0 0 0) 0x7f1d600037a0 con 0x7f1d74101940 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.044+0000 7f1d71ffb640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 0x7f1d640a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.045+0000 7f1d78c51640 1 -- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 msgr2=0x7f1d640a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.045+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7f1d600cedc0 con 0x7f1d74101940 2024-03-18T15:59:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.045+0000 7f1d78c51640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 0x7f1d640a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:59:58.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.049+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d6009be90 con 0x7f1d74101940 2024-03-18T15:59:58.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.232+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1d74062450 con 0x7f1d74101940 2024-03-18T15:59:58.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.236+0000 7f1d71ffb640 1 -- 172.21.15.150:0/4086683980 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7f1d6009b790 con 0x7f1d74101940 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 msgr2=0x7f1d640a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 0x7f1d640a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 msgr2=0x7f1d7419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000e9c0 tx=0x7f1d6000ee80 comp rx=0 tx=0).stop 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 shutdown_connections 2024-03-18T15:59:58.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7f1d640a28f0 0x7f1d640a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d74102300 0x7f1d7419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d74101940 0x7f1d7419c810 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 --2- 172.21.15.150:0/4086683980 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d74100750 0x7f1d7406ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 >> 172.21.15.150:0/4086683980 conn(0x7f1d740fbcc0 msgr2=0x7f1d74104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 shutdown_connections 2024-03-18T15:59:58.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.239+0000 7f1d7aeb5640 1 -- 172.21.15.150:0/4086683980 wait complete. 2024-03-18T15:59:58.250 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:58.179+0000 7f92b9b47200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:59:58.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 -- 172.21.15.150:0/3418192340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830104c10 msgr2=0x7fb830107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 --2- 172.21.15.150:0/3418192340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830104c10 0x7fb830107000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb824009900 tx=0x7fb82402bc30 comp rx=0 tx=0).stop 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 -- 172.21.15.150:0/3418192340 shutdown_connections 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 --2- 172.21.15.150:0/3418192340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb830107540 0x7fb830109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 --2- 172.21.15.150:0/3418192340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830104c10 0x7fb830107000 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 --2- 172.21.15.150:0/3418192340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8301022e0 0x7fb8301046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 -- 172.21.15.150:0/3418192340 >> 172.21.15.150:0/3418192340 conn(0x7fb8300fbc70 msgr2=0x7fb8300fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 -- 172.21.15.150:0/3418192340 shutdown_connections 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.357+0000 7fb8369b2640 1 -- 172.21.15.150:0/3418192340 wait complete. 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 Processor -- start 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 -- start start 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8301022e0 0x7fb830197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb830104c10 0x7fb830198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 0x7fb83019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d450 con 0x7fb830107540 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d2d0 con 0x7fb8301022e0 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb8369b2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d5d0 con 0x7fb830104c10 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 0x7fb83019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 0x7fb83019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44504/0 (socket says 172.21.15.150:44504) 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 -- 172.21.15.150:0/645553794 learned_addr learned my addr 172.21.15.150:0/645553794 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8301022e0 0x7fb830197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb82f7fe640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb830104c10 0x7fb830198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb830104c10 msgr2=0x7fb830198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb830104c10 0x7fb830198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8301022e0 msgr2=0x7fb830197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8301022e0 0x7fb830197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.358+0000 7fb834f4f640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83019f6e0 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb834f4f640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 0x7fb83019f070 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb828009fd0 tx=0x7fb82800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828017070 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83019f9d0 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb828003e50 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8280106c0 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.359+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb83019fef0 con 0x7fb830107540 2024-03-18T15:59:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.360+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8300fd760 con 0x7fb830107540 2024-03-18T15:59:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.362+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) ==== 137308+0+0 (secure 0 0 0) 0x7fb828003970 con 0x7fb830107540 2024-03-18T15:59:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.362+0000 7fb82d7fa640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.362+0000 7fb82ffff640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.363+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8182+0+0 (secure 0 0 0) 0x7fb8280ce700 con 0x7fb830107540 2024-03-18T15:59:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.363+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:59:58.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.365+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb82809b180 con 0x7fb830107540 2024-03-18T15:59:58.518 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:58.248+0000 7f2daaf3e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:59:58.518 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:58.341+0000 7f2daaf3e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:59:58.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.564+0000 7fb82ffff640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:58.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.564+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:59:58.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.572+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb830061c20 con 0x7fb8180a28f0 2024-03-18T15:59:58.589 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:58.277+0000 7f92b9b47200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:59:58.589 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:58.215+0000 7ffb6175f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:59:58.589 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:58.336+0000 7ffb6175f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:59:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-mon[31946]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:59:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-mon[31946]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T15:59:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-mon[31946]: from='client.? ' entity='client.admin' 2024-03-18T15:59:58.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4086683980' entity='client.admin' 2024-03-18T15:59:58.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-mon[36784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:59:58.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-mon[36784]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T15:59:58.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-mon[36784]: from='client.? ' entity='client.admin' 2024-03-18T15:59:58.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 15:59:58 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4086683980' entity='client.admin' 2024-03-18T15:59:58.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.965+0000 7fb82ffff640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:58.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:58.965+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:59:58.992 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-mon[35919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-18T15:59:58.992 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-mon[35919]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-18T15:59:58.992 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-mon[35919]: from='client.? ' entity='client.admin' 2024-03-18T15:59:58.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4086683980' entity='client.admin' 2024-03-18T15:59:59.259 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:58.929+0000 7f92b9b47200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:59:59.260 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.023+0000 7f92b9b47200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:59:59.260 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.164+0000 7f92b9b47200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:59:59.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:58.991+0000 7f2daaf3e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:59:59.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.087+0000 7f2daaf3e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:59:59.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.230+0000 7f2daaf3e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:59:59.380 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.009+0000 7ffb6175f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:59:59.380 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.103+0000 7ffb6175f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:59:59.380 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.249+0000 7ffb6175f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:59:59.592 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.258+0000 7f92b9b47200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:59:59.592 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.341+0000 7f92b9b47200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:59:59.681 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.326+0000 7f2daaf3e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:59:59.681 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.410+0000 7f2daaf3e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:59:59.682 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.680+0000 7f2daaf3e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:59:59.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.378+0000 7ffb6175f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:59:59.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.465+0000 7ffb6175f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:59:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:59.766+0000 7fb82ffff640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T15:59:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T15:59:59.766+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:59:59.894 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.604+0000 7f92b9b47200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:59:59.895 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.714+0000 7f92b9b47200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:59:59.895 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.799+0000 7f92b9b47200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:59:59.895 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 15:59:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T15:59:59.893+0000 7f92b9b47200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:59:59.977 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.792+0000 7f2daaf3e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:59:59.978 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.879+0000 7f2daaf3e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:59:59.978 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 15:59:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T15:59:59.976+0000 7f2daaf3e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:00:00.052 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.795+0000 7ffb6175f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T16:00:00.101 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.907+0000 7ffb6175f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:00.101 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 15:59:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T15:59:59.994+0000 7ffb6175f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T16:00:00.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:00 smithi156 ceph-mon[35919]: overall HEALTH_OK 2024-03-18T16:00:00.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:00.156+0000 7f2daaf3e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:00:00.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:00 smithi150 ceph-mon[31946]: overall HEALTH_OK 2024-03-18T16:00:00.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:00.070+0000 7f92b9b47200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:00:00.462 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:00 smithi191 ceph-mon[36784]: overall HEALTH_OK 2024-03-18T16:00:00.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:00.091+0000 7ffb6175f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:00:00.463 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:00.271+0000 7ffb6175f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:00:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.026+0000 7f92b9b47200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:00:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.155+0000 7f92b9b47200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:00:01.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:01.367+0000 7fb82ffff640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:6800/3095497318 2024-03-18T16:00:01.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:01.367+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T16:00:01.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:01.291+0000 7ffb6175f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:00:01.712 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:01.424+0000 7ffb6175f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:00:01.768 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:01.289+0000 7f2daaf3e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:00:01.768 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:01.421+0000 7f2daaf3e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:00:01.779 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.492+0000 7f92b9b47200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:00:01.779 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.581+0000 7f92b9b47200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:00:02.071 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:01.764+0000 7ffb6175f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:00:02.071 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:01.854+0000 7ffb6175f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:00:02.092 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.778+0000 7f92b9b47200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:02.092 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:01.964+0000 7f92b9b47200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:00:02.105 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:01.803+0000 7f2daaf3e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:00:02.105 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:01.895+0000 7f2daaf3e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:00:02.105 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:02.104+0000 7f2daaf3e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:02.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:02.069+0000 7ffb6175f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:02.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:02.258+0000 7ffb6175f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:00:02.515 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:02.254+0000 7f92b9b47200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:02.518 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:02.319+0000 7f2daaf3e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:00:02.817 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:02.548+0000 7ffb6175f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:02.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:02.514+0000 7f92b9b47200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:02.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:02.753+0000 7f92b9b47200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:00:02.879 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:02.609+0000 7f2daaf3e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:03.212 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:02.815+0000 7ffb6175f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:03.212 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:03.056+0000 7ffb6175f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:00:03.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:02.878+0000 7f2daaf3e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:03.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:03.119+0000 7f2daaf3e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:00:03.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:03 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:03.559+0000 7f92b9b47200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:00:04.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:03.943+0000 7f2daaf3e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:00:04.395 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.130+0000 7f92b9b47200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:00:04.395 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.233+0000 7f92b9b47200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:00:04.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:04.033+0000 7ffb6175f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:00:04.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:04.479+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 20) ==== 136975+0+0 (secure 0 0 0) 0x7fb8280977f0 con 0x7fb830107540 2024-03-18T16:00:04.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:04.480+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 msgr2=0x7fb8180a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:00:04.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:04.480+0000 7fb82d7fa640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/3095497318,v1:172.21.15.150:6801/3095497318] conn(0x7fb8180a28f0 0x7fb8180a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:04.715 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.454+0000 7f92b9b47200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: Active manager daemon x restarted 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: Activating manager daemon x 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: mgrmap e20: x(active, starting, since 0.0125159s), standbys: z, y 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:00:04.716 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:04.717 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T16:00:04.718 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T16:00:04.719 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-mon[31946]: Manager daemon x is now available 2024-03-18T16:00:04.857 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:04.750+0000 7ffb6175f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:00:04.857 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: Active manager daemon x restarted 2024-03-18T16:00:04.857 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: Activating manager daemon x 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: mgrmap e20: x(active, starting, since 0.0125159s), standbys: z, y 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:00:04.858 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:00:04.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T16:00:04.860 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-mon[36784]: Manager daemon x is now available 2024-03-18T16:00:04.896 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:04.536+0000 7f2daaf3e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:00:04.896 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:04.643+0000 7f2daaf3e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:00:04.896 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: Active manager daemon x restarted 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: Activating manager daemon x 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: mgrmap e20: x(active, starting, since 0.0125159s), standbys: z, y 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:00:04.897 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:00:04.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T16:00:04.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-mon[35919]: Manager daemon x is now available 2024-03-18T16:00:05.092 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f921c4dc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.092 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f921c4dc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.092 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f921c4dc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f921c4dc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f921c4dc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f92194d6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f92194d6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f92194d6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f92194d6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.093 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:00:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:00:04.730+0000 7f92194d6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:00:05.145 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:04.855+0000 7ffb6175f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:00:05.268 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:00:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y[37301]: 2024-03-18T16:00:04.894+0000 7f2daaf3e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:00:05.462 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z[38638]: 2024-03-18T16:00:05.144+0000 7ffb6175f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:00:05.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.485+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mon.0 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fb828096ed0 con 0x7fb830107540 2024-03-18T16:00:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.486+0000 7fb82d7fa640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb818051310 0x7fb8180516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:05.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.486+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb828004e50 con 0x7fb818051310 2024-03-18T16:00:05.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.488+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb818051310 0x7fb8180516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:05.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.492+0000 7fb82ffff640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb818051310 0x7fb8180516f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb820005d00 tx=0x7fb82000b410 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:05.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.557+0000 7fb82d7fa640 1 -- 172.21.15.150:0/645553794 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb828004e50 con 0x7fb818051310 2024-03-18T16:00:05.559 INFO:teuthology.orchestra.run.smithi150.stdout:Schedule set for path / 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.559+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb818051310 msgr2=0x7fb8180516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.559+0000 7fb8369b2640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb818051310 0x7fb8180516f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb820005d00 tx=0x7fb82000b410 comp rx=0 tx=0).stop 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 msgr2=0x7fb83019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 --2- 172.21.15.150:0/645553794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb830107540 0x7fb83019f070 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb828009fd0 tx=0x7fb82800b260 comp rx=0 tx=0).stop 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb834f4f640 1 -- 172.21.15.150:0/645553794 reap_dead start 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 shutdown_connections 2024-03-18T16:00:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 >> 172.21.15.150:0/645553794 conn(0x7fb8300fbc70 msgr2=0x7fb830102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:05.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 shutdown_connections 2024-03-18T16:00:05.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.560+0000 7fb8369b2640 1 -- 172.21.15.150:0/645553794 wait complete. 2024-03-18T16:00:05.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.678+0000 7f66a4827640 1 -- 172.21.15.150:0/2919778080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a01018f0 msgr2=0x7f66a0101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.678+0000 7f66a4827640 1 --2- 172.21.15.150:0/2919778080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a01018f0 0x7f66a0101d70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6694009900 tx=0x7f669402bc30 comp rx=0 tx=0).stop 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 -- 172.21.15.150:0/2919778080 shutdown_connections 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 --2- 172.21.15.150:0/2919778080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01022b0 0x7f66a0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 --2- 172.21.15.150:0/2919778080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a01018f0 0x7f66a0101d70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 --2- 172.21.15.150:0/2919778080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a0100700 0x7f66a0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 -- 172.21.15.150:0/2919778080 >> 172.21.15.150:0/2919778080 conn(0x7f66a00fbc70 msgr2=0x7f66a00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 -- 172.21.15.150:0/2919778080 shutdown_connections 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 -- 172.21.15.150:0/2919778080 wait complete. 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 Processor -- start 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 -- start start 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.679+0000 7f66a4827640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a0100700 0x7f66a0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a01022b0 0x7f66a01a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010c380 con 0x7f66a0100700 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010c200 con 0x7f66a01018f0 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010c500 con 0x7f66a01022b0 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:05.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a0100700 0x7f66a0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669ed9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a01022b0 0x7f66a01a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:58728/0 (socket says 172.21.15.150:58728) 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 -- 172.21.15.150:0/784228511 learned_addr learned my addr 172.21.15.150:0/784228511 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 -- 172.21.15.150:0/784228511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a01022b0 msgr2=0x7f66a01a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a01022b0 0x7f66a01a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 -- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a0100700 msgr2=0x7f66a0199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a0100700 0x7f66a0199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 -- 172.21.15.150:0/784228511 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a01a18e0 con 0x7f66a01018f0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f669dd9b640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6694009230 tx=0x7f669402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694041070 con 0x7f66a01018f0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6694004450 con 0x7f66a01018f0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a01a1b70 con 0x7f66a01018f0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694003e20 con 0x7f66a01018f0 2024-03-18T16:00:05.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.680+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a01a20b0 con 0x7f66a01018f0 2024-03-18T16:00:05.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.681+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a01a2280 con 0x7f66a01018f0 2024-03-18T16:00:05.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.682+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f6694044070 con 0x7f66a01018f0 2024-03-18T16:00:05.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.683+0000 7f668f7fe640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 0x7f66880a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:05.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.683+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f66940f0230 con 0x7f66a01018f0 2024-03-18T16:00:05.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.683+0000 7f669e59c640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 0x7f66880a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:05.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.683+0000 7f669e59c640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 0x7f66880a4bf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66a0069a50 tx=0x7f6690009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:05.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.687+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66940bcea0 con 0x7f66a01018f0 2024-03-18T16:00:05.800 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.800 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.800 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T16:00:05.800 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T16:00:05.800 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: Standby manager daemon y restarted 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: Standby manager daemon y started 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: Standby manager daemon z restarted 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: Standby manager daemon z started 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.801 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:05 smithi191 ceph-mon[36784]: mgrmap e21: x(active, since 1.0204s), standbys: y, z 2024-03-18T16:00:05.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: Standby manager daemon y restarted 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: Standby manager daemon y started 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: Standby manager daemon z restarted 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: Standby manager daemon z started 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:05.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.832 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:05.832 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.832 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:05 smithi156 ceph-mon[35919]: mgrmap e21: x(active, since 1.0204s), standbys: y, z 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: Standby manager daemon y restarted 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: Standby manager daemon y started 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.156:0/1692664676' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: Standby manager daemon z restarted 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: Standby manager daemon z started 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: from='mgr.? 172.21.15.191:0/886753344' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:05.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:05 smithi150 ceph-mon[31946]: mgrmap e21: x(active, since 1.0204s), standbys: y, z 2024-03-18T16:00:05.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.895+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f66a00fde30 con 0x7f66880a2730 2024-03-18T16:00:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.950+0000 7f668f7fe640 1 -- 172.21.15.150:0/784228511 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f66a00fde30 con 0x7f66880a2730 2024-03-18T16:00:05.952 INFO:teuthology.orchestra.run.smithi150.stdout:Retention added to path / 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 msgr2=0x7f66880a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 0x7f66880a4bf0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66a0069a50 tx=0x7f6690009500 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 msgr2=0x7f66a019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6694009230 tx=0x7f669402c650 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 shutdown_connections 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f66880a2730 0x7f66880a4bf0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66a01022b0 0x7f66a01a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66a01018f0 0x7f66a019a230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 --2- 172.21.15.150:0/784228511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66a0100700 0x7f66a0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 >> 172.21.15.150:0/784228511 conn(0x7f66a00fbc70 msgr2=0x7f66a00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:05.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 shutdown_connections 2024-03-18T16:00:05.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:05.952+0000 7f66a4827640 1 -- 172.21.15.150:0/784228511 wait complete. 2024-03-18T16:00:05.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 -- 172.21.15.150:0/2942678968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94102a00 msgr2=0x7f8a94102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/2942678968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94102a00 0x7f8a94102e80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009900 tx=0x7f8a8402bc30 comp rx=0 tx=0).stop 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 -- 172.21.15.150:0/2942678968 shutdown_connections 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/2942678968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a941033c0 0x7f8a94109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/2942678968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94102a00 0x7f8a94102e80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/2942678968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a94101810 0x7f8a94101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 -- 172.21.15.150:0/2942678968 >> 172.21.15.150:0/2942678968 conn(0x7f8a940fcda0 msgr2=0x7f8a940ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 -- 172.21.15.150:0/2942678968 shutdown_connections 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.068+0000 7f8a9c37e640 1 -- 172.21.15.150:0/2942678968 wait complete. 2024-03-18T16:00:06.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 Processor -- start 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 -- start start 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a94102a00 0x7f8a94197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a941033c0 0x7f8a9419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d490 con 0x7f8a94101810 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d310 con 0x7f8a941033c0 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9c37e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d610 con 0x7f8a94102a00 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53578/0 (socket says 172.21.15.150:53578) 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 -- 172.21.15.150:0/3809531639 learned_addr learned my addr 172.21.15.150:0/3809531639 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a99919640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a94102a00 0x7f8a94197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a91b640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a941033c0 0x7f8a9419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 -- 172.21.15.150:0/3809531639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a94102a00 msgr2=0x7f8a94197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a94102a00 0x7f8a94197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 -- 172.21.15.150:0/3809531639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a941033c0 msgr2=0x7f8a9419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a941033c0 0x7f8a9419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.069+0000 7f8a9a11a640 1 -- 172.21.15.150:0/3809531639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9419f640 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a9a11a640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e9c0 tx=0x7f8a9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000cd80 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a900035f0 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90010690 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9419f930 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.070+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9419fe50 con 0x7f8a94101810 2024-03-18T16:00:06.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.071+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a94108a60 con 0x7f8a94101810 2024-03-18T16:00:06.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.072+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f8a900037a0 con 0x7f8a94101810 2024-03-18T16:00:06.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.072+0000 7f8a8b7fe640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 0x7f8a800a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.073+0000 7f8a99919640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 0x7f8a800a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.073+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8a900ceae0 con 0x7f8a94101810 2024-03-18T16:00:06.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.073+0000 7f8a99919640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 0x7f8a800a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402c7b0 tx=0x7f8a84005fd0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:06.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.077+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a9009bda0 con 0x7f8a94101810 2024-03-18T16:00:06.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.297+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8a94061c20 con 0x7f8a800a26b0 2024-03-18T16:00:06.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.305+0000 7f8a8b7fe640 1 -- 172.21.15.150:0/3809531639 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f8a94061c20 con 0x7f8a800a26b0 2024-03-18T16:00:06.307 INFO:teuthology.orchestra.run.smithi150.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T16:00:05", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.307+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 msgr2=0x7f8a800a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.307+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 0x7f8a800a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a8402c7b0 tx=0x7f8a84005fd0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.307+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 msgr2=0x7f8a94197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.307+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e9c0 tx=0x7f8a9000ee80 comp rx=0 tx=0).stop 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 shutdown_connections 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8a800a26b0 0x7f8a800a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a941033c0 0x7f8a9419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a94102a00 0x7f8a94197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 --2- 172.21.15.150:0/3809531639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a94101810 0x7f8a94197a20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 >> 172.21.15.150:0/3809531639 conn(0x7f8a940fcda0 msgr2=0x7f8a94105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:06.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 shutdown_connections 2024-03-18T16:00:06.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.308+0000 7f8a9c37e640 1 -- 172.21.15.150:0/3809531639 wait complete. 2024-03-18T16:00:06.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T16:00:06.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 -- 172.21.15.150:0/1332326317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f781022d0 msgr2=0x7f1f78108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 --2- 172.21.15.150:0/1332326317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f781022d0 0x7f1f78108b60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009ca0 tx=0x7f1f7002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 -- 172.21.15.150:0/1332326317 shutdown_connections 2024-03-18T16:00:06.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 --2- 172.21.15.150:0/1332326317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f781022d0 0x7f1f78108b60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 --2- 172.21.15.150:0/1332326317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f78101910 0x7f1f78101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 --2- 172.21.15.150:0/1332326317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 0x7f1f78100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 -- 172.21.15.150:0/1332326317 >> 172.21.15.150:0/1332326317 conn(0x7f1f780fbc70 msgr2=0x7f1f780fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 -- 172.21.15.150:0/1332326317 shutdown_connections 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.433+0000 7f1f7e787640 1 -- 172.21.15.150:0/1332326317 wait complete. 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 Processor -- start 2024-03-18T16:00:06.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 -- start start 2024-03-18T16:00:06.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 0x7f1f78196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f781022d0 0x7f1f7819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810c3a0 con 0x7f1f78101910 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810c220 con 0x7f1f781022d0 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.434+0000 7f1f7e787640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810c520 con 0x7f1f78100720 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f77fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 0x7f1f78196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53592/0 (socket says 172.21.15.150:53592) 2024-03-18T16:00:06.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 -- 172.21.15.150:0/3107908053 learned_addr learned my addr 172.21.15.150:0/3107908053 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f7cd24640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f781022d0 0x7f1f7819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 -- 172.21.15.150:0/3107908053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 msgr2=0x7f1f78196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 0x7f1f78196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 -- 172.21.15.150:0/3107908053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f781022d0 msgr2=0x7f1f7819dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f781022d0 0x7f1f7819dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 -- 172.21.15.150:0/3107908053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7819e550 con 0x7f1f78101910 2024-03-18T16:00:06.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f777fe640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e9f0 tx=0x7f1f6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00cd80 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c010910 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7819e840 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c018990 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.435+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7819ede0 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.436+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f781079e0 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.437+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f1f6c010ac0 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.438+0000 7f1f757fa640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 0x7f1f600a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.438+0000 7f1f77fff640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 0x7f1f600a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.439+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f1f6c0cee00 con 0x7f1f78101910 2024-03-18T16:00:06.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.439+0000 7f1f77fff640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 0x7f1f600a4bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1f78101770 tx=0x7f1f68005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:06.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.442+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f6c09ba30 con 0x7f1f78101910 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: [18/Mar/2024:16:00:04] ENGINE Bus STARTING 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: [18/Mar/2024:16:00:05] ENGINE Serving on https://172.21.15.150:7150 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: [18/Mar/2024:16:00:05] ENGINE Serving on http://172.21.15.150:8765 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: [18/Mar/2024:16:00:05] ENGINE Bus STARTED 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:06.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:06 smithi150 ceph-mon[31946]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:06.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.690+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1f78061c20 con 0x7f1f600a2730 2024-03-18T16:00:06.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.697+0000 7f1f757fa640 1 -- 172.21.15.150:0/3107908053 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1f78061c20 con 0x7f1f600a2730 2024-03-18T16:00:06.698 INFO:teuthology.orchestra.run.smithi150.stdout:/ 1m 6m3h 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.699+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 msgr2=0x7f1f600a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.699+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 0x7f1f600a4bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1f78101770 tx=0x7f1f68005f90 comp rx=0 tx=0).stop 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 msgr2=0x7f1f78196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00e9f0 tx=0x7f1f6c00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 shutdown_connections 2024-03-18T16:00:06.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f1f600a2730 0x7f1f600a4bf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f781022d0 0x7f1f7819dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f78101910 0x7f1f78196e70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 --2- 172.21.15.150:0/3107908053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f78100720 0x7f1f78196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 >> 172.21.15.150:0/3107908053 conn(0x7f1f780fbc70 msgr2=0x7f1f78104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 shutdown_connections 2024-03-18T16:00:06.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:00:06.700+0000 7f1f7e787640 1 -- 172.21.15.150:0/3107908053 wait complete. 2024-03-18T16:00:06.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-18T16:00:06.742 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-18T16:00:06.752 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-18T16:00:06.800 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T16:00:06.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-18T16:00:06.802 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T16:00:06.815 INFO:tasks.workunit:Pulling workunits from ref 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T16:00:06.815 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T16:00:06.815 INFO:tasks.workunit:timeout=3h 2024-03-18T16:00:06.815 INFO:tasks.workunit:cleanup=True 2024-03-18T16:00:06.816 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-18 15:58:02.587749954 +0000 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-18 15:58:02.587749954 +0000 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-18 15:59:55.481309396 +0000 2024-03-18T16:00:06.831 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-18 15:58:02.587749954 +0000 2024-03-18T16:00:06.832 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T16:00:06.832 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T16:00:06.906 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T16:00:06.961 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: [18/Mar/2024:16:00:04] ENGINE Bus STARTING 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: [18/Mar/2024:16:00:05] ENGINE Serving on https://172.21.15.150:7150 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: [18/Mar/2024:16:00:05] ENGINE Serving on http://172.21.15.150:8765 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: [18/Mar/2024:16:00:05] ENGINE Bus STARTED 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:06.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:06 smithi191 ceph-mon[36784]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: [18/Mar/2024:16:00:04] ENGINE Bus STARTING 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: [18/Mar/2024:16:00:05] ENGINE Serving on https://172.21.15.150:7150 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: [18/Mar/2024:16:00:05] ENGINE Serving on http://172.21.15.150:8765 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: [18/Mar/2024:16:00:05] ENGINE Bus STARTED 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:07.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:06 smithi156 ceph-mon[35919]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:07 smithi191 ceph-mon[36784]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-18T16:00:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:07 smithi156 ceph-mon[35919]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='client.25172 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:07 smithi150 ceph-mon[31946]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-18T16:00:08.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:08.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:08 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:09.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:08 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:09.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:09.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:08 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:00:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:09.965 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:00:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:10.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:00:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi150:/etc/ceph/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi191:/etc/ceph/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:10.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:00:10.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:10 smithi191 ceph-mon[36784]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:10.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:10 smithi191 ceph-mon[36784]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:10.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:10 smithi191 ceph-mon[36784]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:10 smithi156 ceph-mon[35919]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:10 smithi156 ceph-mon[35919]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:10 smithi156 ceph-mon[35919]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:10 smithi150 ceph-mon[31946]: Updating smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:10 smithi150 ceph-mon[31946]: Updating smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:10 smithi150 ceph-mon[31946]: Updating smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T16:00:11.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:11 smithi150 ceph-mon[31946]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:11 smithi191 ceph-mon[36784]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:11 smithi156 ceph-mon[35919]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:00:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:13 smithi191 ceph-mon[36784]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-18T16:00:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:13 smithi156 ceph-mon[35919]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-18T16:00:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:13 smithi150 ceph-mon[31946]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-18T16:00:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:15 smithi191 ceph-mon[36784]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T16:00:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:15 smithi156 ceph-mon[35919]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T16:00:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:15 smithi150 ceph-mon[31946]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-18T16:00:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:17 smithi191 ceph-mon[36784]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T16:00:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:17 smithi156 ceph-mon[35919]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T16:00:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:17 smithi150 ceph-mon[31946]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 -- 172.21.15.150:0/1058440366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0364069f80 msgr2=0x7f036406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 --2- 172.21.15.150:0/1058440366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0364069f80 0x7f036406a3e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f034c009930 tx=0x7f034c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 -- 172.21.15.150:0/1058440366 shutdown_connections 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 --2- 172.21.15.150:0/1058440366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f036406a920 0x7f036410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 --2- 172.21.15.150:0/1058440366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0364069f80 0x7f036406a3e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 --2- 172.21.15.150:0/1058440366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03641067b0 0x7f0364106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 -- 172.21.15.150:0/1058440366 >> 172.21.15.150:0/1058440366 conn(0x7f03640776c0 msgr2=0x7f0364077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 -- 172.21.15.150:0/1058440366 shutdown_connections 2024-03-18T16:00:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.935+0000 7f0369732640 1 -- 172.21.15.150:0/1058440366 wait complete. 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 Processor -- start 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 -- start start 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f036406a920 0x7f03641984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 0x7f036419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0364111b60 con 0x7f03641067b0 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f03641119e0 con 0x7f036406a920 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0369732640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0364111ce0 con 0x7f0364069f80 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f03637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 0x7f036419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:36896/0 (socket says 172.21.15.150:36896) 2024-03-18T16:00:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f035bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f036406a920 0x7f03641984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 -- 172.21.15.150:0/3796400461 learned_addr learned my addr 172.21.15.150:0/3796400461 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 -- 172.21.15.150:0/3796400461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f036406a920 msgr2=0x7f03641984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f036406a920 0x7f03641984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 -- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 msgr2=0x7f036419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 0x7f036419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f0362ffd640 1 -- 172.21.15.150:0/3796400461 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036419cf90 con 0x7f0364069f80 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.936+0000 7f03637fe640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 0x7f036419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0362ffd640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f035400e9f0 tx=0x7f035400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0360ff9640 1 -- 172.21.15.150:0/3796400461 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035400cd80 con 0x7f0364069f80 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036419d280 con 0x7f0364069f80 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0360ff9640 1 -- 172.21.15.150:0/3796400461 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0354010910 con 0x7f0364069f80 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0360ff9640 1 -- 172.21.15.150:0/3796400461 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03540188f0 con 0x7f0364069f80 2024-03-18T16:00:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.937+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03641a4b10 con 0x7f0364069f80 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.939+0000 7f0360ff9640 1 -- 172.21.15.150:0/3796400461 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 23) ==== 137467+0+0 (secure 0 0 0) 0x7f0354010ac0 con 0x7f0364069f80 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.939+0000 7f036c3f4740 1 Processor -- start 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.939+0000 7f036c3f4740 1 -- start start 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.939+0000 7f0360ff9640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 0x7f03500a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.939+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f036c3f4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5621a2131a40 con 0x5621a21cd130 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f036c3f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5621a20da920 con 0x5621a21d6820 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f036c3f4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5621a212dae0 con 0x5621a21cda50 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03637fe640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 0x7f03500a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49292/0 (socket says 172.21.15.150:49292) 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f0360ff9640 1 -- 172.21.15.150:0/3796400461 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f03540cf3b0 con 0x7f0364069f80 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f0362ffd640 1 -- 172.21.15.150:0/2124386283 learned_addr learned my addr 172.21.15.150:0/2124386283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03637fe640 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 -- 172.21.15.150:0/2124386283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 msgr2=0x5621a21d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21d8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 -- 172.21.15.150:0/2124386283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 msgr2=0x5621a21cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 -- 172.21.15.150:0/2124386283 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a20f2460 con 0x5621a21cda50 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03637fe640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 0x7f03500a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f035c005fd0 tx=0x7f035c013660 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f035bfff640 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21d62e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f034c009230 tx=0x7f034c009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffb4a782dbf44d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03597fa640 1 -- 172.21.15.150:0/2124386283 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c042070 con 0x5621a21cda50 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03597fa640 1 -- 172.21.15.150:0/2124386283 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034c004030 con 0x5621a21cda50 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.940+0000 7f03597fa640 1 -- 172.21.15.150:0/2124386283 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c004b20 con 0x5621a21cda50 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- 172.21.15.150:0/2124386283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 msgr2=0x5621a21d62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21d62e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f034c009230 tx=0x7f034c009610 comp rx=0 tx=0).stop 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- 172.21.15.150:0/2124386283 shutdown_connections 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21d62e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2124386283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- 172.21.15.150:0/2124386283 >> 172.21.15.150:0/2124386283 conn(0x5621a21c7a50 msgr2=0x5621a21c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- 172.21.15.150:0/2124386283 shutdown_connections 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- 172.21.15.150:0/2124386283 wait complete. 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.941+0000 7f036c3f4740 1 -- start start 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 Processor -- start 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cda50 0x5621a21d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a21d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5621a21d93f0 con 0x5621a21cd130 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5621a20edd70 con 0x5621a21cda50 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5621a21d9570 con 0x5621a21d6820 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f036c3f4740 10 client.0 _ll_register_callbacks cb 0x5621a1e45a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49306/0 (socket says 172.21.15.150:49306) 2024-03-18T16:00:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 -- 172.21.15.150:0/2693909849 learned_addr learned my addr 172.21.15.150:0/2693909849 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f035bfff640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cda50 0x5621a21d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f03637fe640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a21d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 -- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 msgr2=0x5621a21d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a21d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 -- 172.21.15.150:0/2693909849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cda50 msgr2=0x5621a21d3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cda50 0x5621a21d3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 -- 172.21.15.150:0/2693909849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a1e46210 con 0x5621a21cd130 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f035409b8d0 tx=0x7f03540147e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbfb53cbf2ad4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f034a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.942+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035409b940 con 0x5621a21cd130 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f03540ab5c0 con 0x5621a21cd130 2024-03-18T16:00:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034c004030 con 0x5621a21cd130 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f035409f9e0 con 0x5621a21cd130 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 10 client.15426 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5621a1f12620 con 0x5621a21cd130 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f034c004030 con 0x5621a21cd130 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f035409cbc0 con 0x5621a21cd130 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 10 client.15426 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 10 client.15426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 20 client.15426 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a21e69d0 0x5621a21e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 4 client.15426 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621a213ef80 con 0x5621a21e69d0 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.943+0000 7f035bfff640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a21e69d0 0x5621a21e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.944+0000 7f034a7fc640 10 client.15426 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.944+0000 7f035bfff640 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a21e69d0 0x5621a21e8dc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6efd55ce3824f1a3 server_cookie=f164d88f61fbd1ba in_seq=0 out_seq=0 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f034a7fc640 1 -- 172.21.15.150:0/2693909849 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621a213ef80 con 0x5621a21e69d0 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f034a7fc640 10 client.15426 handle_command_reply: tid=0 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 client.15426 shutdown 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 msgr2=0x5621a21d38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f035409b8d0 tx=0x7f03540147e0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 shutdown_connections 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a21e69d0 0x5621a21e8dc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a21d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cda50 0x5621a21d3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 --2- 172.21.15.150:0/2693909849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d38e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f034a7fc640 0 client.15426 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 >> 172.21.15.150:0/2693909849 conn(0x5621a21c7a50 msgr2=0x5621a21c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 shutdown_connections 2024-03-18T16:00:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 1 -- 172.21.15.150:0/2693909849 wait complete. 2024-03-18T16:00:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.945+0000 7f036c3f4740 20 client.15426 trim_cache size 0 max 16384 2024-03-18T16:00:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.959+0000 7f036c3f4740 1 Processor -- start 2024-03-18T16:00:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.959+0000 7f036c3f4740 1 -- start start 2024-03-18T16:00:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21e2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03641afc10 con 0x5621a21cd130 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f03641ad4e0 con 0x5621a21d6820 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f036c3f4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f03641ab330 con 0x5621a21cda50 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49322/0 (socket says 172.21.15.150:49322) 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 -- 172.21.15.150:0/1535360219 learned_addr learned my addr 172.21.15.150:0/1535360219 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f035bfff640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21e2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f03637fe640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 -- 172.21.15.150:0/1535360219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 msgr2=0x5621a21e2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21e2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 -- 172.21.15.150:0/1535360219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 msgr2=0x5621a21ee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21ee6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 -- 172.21.15.150:0/1535360219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a21ce060 con 0x5621a21cd130 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f035bfff640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21e2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0362ffd640 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f03540a3b00 tx=0x7f035400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3393c2b5858cea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0358ff9640 1 -- 172.21.15.150:0/1535360219 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03540af040 con 0x5621a21cd130 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0358ff9640 1 -- 172.21.15.150:0/1535360219 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03540a7070 con 0x5621a21cd130 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.960+0000 7f0358ff9640 1 -- 172.21.15.150:0/1535360219 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03540a3ba0 con 0x5621a21cd130 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- 172.21.15.150:0/1535360219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 msgr2=0x5621a21d6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f03540a3b00 tx=0x7f035400cf20 comp rx=0 tx=0).stop 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- 172.21.15.150:0/1535360219 shutdown_connections 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21d6820 0x5621a21ee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21cda50 0x5621a21e2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 --2- 172.21.15.150:0/1535360219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cd130 0x5621a21d6010 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- 172.21.15.150:0/1535360219 >> 172.21.15.150:0/1535360219 conn(0x5621a21c7a50 msgr2=0x5621a21c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- 172.21.15.150:0/1535360219 shutdown_connections 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- 172.21.15.150:0/1535360219 wait complete. 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 -- start start 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.961+0000 7f036c3f4740 1 Processor -- start 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a224a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03641ab9a0 con 0x5621a21cda50 2024-03-18T16:00:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f03641ace10 con 0x5621a21cd130 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f03641ac070 con 0x5621a21d6820 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f036c3f4740 10 client.0 _ll_register_callbacks cb 0x7f035409b530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f035bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f035bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49326/0 (socket says 172.21.15.150:49326) 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:50700/0 (socket says 172.21.15.150:50700) 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f035bfff640 1 -- 172.21.15.150:0/873765053 learned_addr learned my addr 172.21.15.150:0/873765053 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f03637fe640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a224a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 -- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 msgr2=0x5621a224a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a224a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 -- 172.21.15.150:0/873765053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 msgr2=0x5621a21d1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 -- 172.21.15.150:0/873765053 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a224a740 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f035bfff640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f0362ffd640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f035409b8d0 tx=0x7f035400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9957b9fc37bb9de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.962+0000 7f034a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03540b3070 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f03540a3da0 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03540a7070 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f03540b8070 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 10 client.25208 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5621a2251af0 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03540a7070 con 0x5621a21cd130 2024-03-18T16:00:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f035409ad90 con 0x5621a21cd130 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 10 client.25208 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 10 client.25208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 20 client.25208 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a2252020 0x5621a2254410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 4 client.25208 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621a21392d0 con 0x5621a2252020 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.963+0000 7f035bfff640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a2252020 0x5621a2254410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.964+0000 7f034a7fc640 10 client.25208 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.964+0000 7f035bfff640 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a2252020 0x5621a2254410 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=845a527cfa542270 server_cookie=48e8b8e229e268b6 in_seq=0 out_seq=0 2024-03-18T16:00:19.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:19 smithi191 ceph-mon[36784]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f034a7fc640 1 -- 172.21.15.150:0/873765053 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621a21392d0 con 0x5621a2252020 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f034a7fc640 10 client.25208 handle_command_reply: tid=0 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 client.25208 shutdown 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 msgr2=0x5621a21d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f035409b8d0 tx=0x7f035400cf90 comp rx=0 tx=0).stop 2024-03-18T16:00:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 shutdown_connections 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5621a2252020 0x5621a2254410 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5621a21d6820 0x5621a224a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f034a7fc640 0 client.25208 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5621a21cda50 0x5621a21d1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 --2- 172.21.15.150:0/873765053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621a21cd130 0x5621a21d0d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 >> 172.21.15.150:0/873765053 conn(0x5621a21c7a50 msgr2=0x5621a21c6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 shutdown_connections 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 1 -- 172.21.15.150:0/873765053 wait complete. 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.974+0000 7f036c3f4740 20 client.25208 trim_cache size 0 max 16384 2024-03-18T16:00:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:00:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:00:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:00:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 msgr2=0x7f03500a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 0x7f03500a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f035c005fd0 tx=0x7f035c013660 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 msgr2=0x7f0364197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f035400e9f0 tx=0x7f035400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 shutdown_connections 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f03500a2ab0 0x7f03500a4f70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641067b0 0x7f036419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f036406a920 0x7f03641984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 --2- 172.21.15.150:0/3796400461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0364069f80 0x7f0364197fa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 >> 172.21.15.150:0/3796400461 conn(0x7f03640776c0 msgr2=0x7f03640fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 shutdown_connections 2024-03-18T16:00:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:19.975+0000 7f0369732640 1 -- 172.21.15.150:0/3796400461 wait complete. 2024-03-18T16:00:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:19 smithi156 ceph-mon[35919]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:19 smithi150 ceph-mon[31946]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:21 smithi191 ceph-mon[36784]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:21 smithi156 ceph-mon[35919]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:21 smithi150 ceph-mon[31946]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-18T16:00:23.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:23 smithi191 ceph-mon[36784]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-18T16:00:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:23 smithi156 ceph-mon[35919]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-18T16:00:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:23 smithi150 ceph-mon[31946]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-18T16:00:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:25 smithi191 ceph-mon[36784]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:25 smithi156 ceph-mon[35919]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:25 smithi150 ceph-mon[31946]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:27 smithi191 ceph-mon[36784]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:27 smithi156 ceph-mon[35919]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:27 smithi150 ceph-mon[31946]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:29 smithi191 ceph-mon[36784]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:29 smithi156 ceph-mon[35919]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:29 smithi150 ceph-mon[31946]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:31.893 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:31 smithi150 ceph-mon[31946]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:31 smithi191 ceph-mon[36784]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:31 smithi156 ceph-mon[35919]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 -- 172.21.15.150:0/2690370314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4394102150 msgr2=0x7f439410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 --2- 172.21.15.150:0/2690370314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4394102150 0x7f439410e660 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f438c009c10 tx=0x7f438c02bef0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 -- 172.21.15.150:0/2690370314 shutdown_connections 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 --2- 172.21.15.150:0/2690370314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4394102150 0x7f439410e660 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 --2- 172.21.15.150:0/2690370314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394101790 0x7f4394101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 --2- 172.21.15.150:0/2690370314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43941077a0 0x7f4394107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 -- 172.21.15.150:0/2690370314 >> 172.21.15.150:0/2690370314 conn(0x7f43940fd2a0 msgr2=0x7f43940ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 -- 172.21.15.150:0/2690370314 shutdown_connections 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.064+0000 7f4398bc6640 1 -- 172.21.15.150:0/2690370314 wait complete. 2024-03-18T16:00:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 Processor -- start 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 -- start start 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4394101790 0x7f4394197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 0x7f439419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111bb0 con 0x7f4394102150 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111a30 con 0x7f4394101790 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4398bc6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111d30 con 0x7f43941077a0 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.065+0000 7f439259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4394101790 0x7f4394197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 0x7f439419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51494/0 (socket says 172.21.15.150:51494) 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4391d9b640 1 -- 172.21.15.150:0/3531935128 learned_addr learned my addr 172.21.15.150:0/3531935128 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 -- 172.21.15.150:0/3531935128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4394101790 msgr2=0x7f4394197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4394101790 0x7f4394197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 -- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 msgr2=0x7f4394198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 -- 172.21.15.150:0/3531935128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439419cfe0 con 0x7f43941077a0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4391d9b640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4392d9d640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 0x7f439419c8b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c430 tx=0x7f438c0336b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f43837fe640 1 -- 172.21.15.150:0/3531935128 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c043020 con 0x7f43941077a0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f43837fe640 1 -- 172.21.15.150:0/3531935128 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c02ca80 con 0x7f43941077a0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439419d270 con 0x7f43941077a0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f43837fe640 1 -- 172.21.15.150:0/3531935128 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c03e6b0 con 0x7f43941077a0 2024-03-18T16:00:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.066+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43941a4b60 con 0x7f43941077a0 2024-03-18T16:00:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.069+0000 7f43837fe640 1 -- 172.21.15.150:0/3531935128 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 23) ==== 137467+0+0 (secure 0 0 0) 0x7f438c046020 con 0x7f43941077a0 2024-03-18T16:00:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.069+0000 7f439b7b3740 1 Processor -- start 2024-03-18T16:00:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.069+0000 7f439b7b3740 1 -- start start 2024-03-18T16:00:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 0x55acaacf5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaacfe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 0x55acaad00c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55acaac59a40 con 0x55acaacf5130 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acaac02920 con 0x55acaacf5a50 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439b7b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55acaac55ae0 con 0x55acaacfe820 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 0x55acaacf5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f43837fe640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 0x7f437c0a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaacfe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 0x55acaacf5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51500/0 (socket says 172.21.15.150:51500) 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4391d9b640 1 -- 172.21.15.150:0/353180861 learned_addr learned my addr 172.21.15.150:0/353180861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4391d9b640 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 0x55acaad00c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4392d9d640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 0x7f437c0a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f43837fe640 1 -- 172.21.15.150:0/3531935128 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f438c0f12c0 con 0x7f43941077a0 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 -- 172.21.15.150:0/353180861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 msgr2=0x55acaad00c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 0x55acaad00c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 -- 172.21.15.150:0/353180861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 msgr2=0x55acaacf5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 0x55acaacf5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 -- 172.21.15.150:0/353180861 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acaac1a460 con 0x55acaacf5a50 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4391d9b640 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 0x55acaad00c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f439259c640 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaacfe2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4384005850 tx=0x7f4384005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcc091791a56b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.070+0000 7f4392d9d640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 0x7f437c0a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f438c0eeb10 tx=0x7f438c02bef0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f4380ff9640 1 -- 172.21.15.150:0/353180861 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384004050 con 0x55acaacf5a50 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f4380ff9640 1 -- 172.21.15.150:0/353180861 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438401a070 con 0x55acaacf5a50 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f4380ff9640 1 -- 172.21.15.150:0/353180861 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384004a90 con 0x55acaacf5a50 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 -- 172.21.15.150:0/353180861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 msgr2=0x55acaacfe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaacfe2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4384005850 tx=0x7f4384005880 comp rx=0 tx=0).stop 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 -- 172.21.15.150:0/353180861 shutdown_connections 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacfe820 0x55acaad00c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaacfe2e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 --2- 172.21.15.150:0/353180861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5130 0x55acaacf5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 -- 172.21.15.150:0/353180861 >> 172.21.15.150:0/353180861 conn(0x55acaacefa50 msgr2=0x55acaacefe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.071+0000 7f439b7b3740 1 -- 172.21.15.150:0/353180861 shutdown_connections 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 -- 172.21.15.150:0/353180861 wait complete. 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 -- start start 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 Processor -- start 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaacfb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacfc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55acaad01450 con 0x55acaacf5a50 2024-03-18T16:00:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55acaac15d70 con 0x55acaacfe820 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55acaad015d0 con 0x55acaacf5130 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439b7b3740 10 client.0 _ll_register_callbacks cb 0x55acaa96da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51510/0 (socket says 172.21.15.150:51510) 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacfc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f439259c640 1 -- 172.21.15.150:0/1054409401 learned_addr learned my addr 172.21.15.150:0/1054409401 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.072+0000 7f4391d9b640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaacfb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 -- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 msgr2=0x55acaacfb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaacfb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 -- 172.21.15.150:0/1054409401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 msgr2=0x55acaacfc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacfc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 -- 172.21.15.150:0/1054409401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acaa96e210 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4391d9b640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacfc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439259c640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4384004650 tx=0x7f43840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79133e59d8db0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4379ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384003ca0 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4384016450 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438401a070 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4384037050 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439b7b3740 10 client.15444 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55acaaa3a620 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.073+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438401a070 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4384029420 con 0x55acaacf5a50 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 10 client.15444 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 10 client.15444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 20 client.15444 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad0e8e0 0x55acaad10cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 4 client.15444 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acaac66f80 con 0x55acaad0e8e0 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f4391d9b640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad0e8e0 0x55acaad10cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f4379ffb640 10 client.15444 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.074+0000 7f4391d9b640 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad0e8e0 0x55acaad10cd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a32406b299bee9 server_cookie=bcd4d3997dcf85c1 in_seq=0 out_seq=0 2024-03-18T16:00:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f4379ffb640 1 -- 172.21.15.150:0/1054409401 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55acaac66f80 con 0x55acaad0e8e0 2024-03-18T16:00:33.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f4379ffb640 10 client.15444 handle_command_reply: tid=0 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 client.15444 shutdown 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 msgr2=0x55acaacfbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4384004650 tx=0x7f43840038a0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 shutdown_connections 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad0e8e0 0x55acaad10cd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacfc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacfbda0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1054409401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaacfb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.075+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 >> 172.21.15.150:0/1054409401 conn(0x55acaacefa50 msgr2=0x55acaacf17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.076+0000 7f4379ffb640 0 client.15444 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.076+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 shutdown_connections 2024-03-18T16:00:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.076+0000 7f439b7b3740 1 -- 172.21.15.150:0/1054409401 wait complete. 2024-03-18T16:00:33.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.076+0000 7f439b7b3740 20 client.15444 trim_cache size 0 max 16384 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 Processor -- start 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 -- start start 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad108f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacfe820 0x55acaad16300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43941ad910 con 0x55acaacfe820 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43941ab1e0 con 0x55acaacf5a50 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.091+0000 7f439b7b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43941a9050 con 0x55acaacf5130 2024-03-18T16:00:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:60986/0 (socket says 172.21.15.150:60986) 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad108f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:59794/0 (socket says 172.21.15.150:59794) 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 -- 172.21.15.150:0/1839144615 learned_addr learned my addr 172.21.15.150:0/1839144615 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f4391d9b640 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacfe820 0x55acaad16300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 -- 172.21.15.150:0/1839144615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 msgr2=0x55acaad108f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad108f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 -- 172.21.15.150:0/1839144615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacfe820 msgr2=0x55acaad16300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacfe820 0x55acaad16300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 -- 172.21.15.150:0/1839144615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acaad00c10 con 0x55acaacf5a50 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f439259c640 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43840298c0 tx=0x7f43840096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b67e41e55c55325 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f437bfff640 1 -- 172.21.15.150:0/1839144615 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384009a50 con 0x55acaacf5a50 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f437bfff640 1 -- 172.21.15.150:0/1839144615 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438401a050 con 0x55acaacf5a50 2024-03-18T16:00:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.092+0000 7f437bfff640 1 -- 172.21.15.150:0/1839144615 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384029420 con 0x55acaacf5a50 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 -- 172.21.15.150:0/1839144615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 msgr2=0x55acaad0a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43840298c0 tx=0x7f43840096a0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 -- 172.21.15.150:0/1839144615 shutdown_connections 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacfe820 0x55acaad16300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacf5a50 0x55acaad0a1f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 --2- 172.21.15.150:0/1839144615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad108f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 -- 172.21.15.150:0/1839144615 >> 172.21.15.150:0/1839144615 conn(0x55acaacefa50 msgr2=0x55acaacfa130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 -- 172.21.15.150:0/1839144615 shutdown_connections 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.093+0000 7f439b7b3740 1 -- 172.21.15.150:0/1839144615 wait complete. 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 -- start start 2024-03-18T16:00:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 Processor -- start 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacf7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacf8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43941a96a0 con 0x55acaacf5a50 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43941aab10 con 0x55acaacfe820 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43941a9d70 con 0x55acaacf5130 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439b7b3740 10 client.0 _ll_register_callbacks cb 0x7f4388013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacf7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:59808/0 (socket says 172.21.15.150:59808) 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 -- 172.21.15.150:0/2343115299 learned_addr learned my addr 172.21.15.150:0/2343115299 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacf8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 -- 172.21.15.150:0/2343115299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 msgr2=0x55acaacf8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacf8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 -- 172.21.15.150:0/2343115299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 msgr2=0x55acaacf7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacf7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f4391d9b640 1 -- 172.21.15.150:0/2343115299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acaad06ee0 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.094+0000 7f439259c640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4391d9b640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4388015010 tx=0x7f43880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03d985cf6c863da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388027db0 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438401a050 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f4388005020 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4388005220 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 10 client.25276 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55acaad74430 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438401a050 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f43880047a0 con 0x55acaacf5130 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 10 client.25276 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:33.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 10 client.25276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 20 client.25276 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad74ac0 0x55acaad76ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 4 client.25276 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55acaad110c0 con 0x55acaad74ac0 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.095+0000 7f439259c640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad74ac0 0x55acaad76ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.096+0000 7f4379ffb640 10 client.25276 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.096+0000 7f439259c640 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad74ac0 0x55acaad76ee0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b139a9763c6562bc server_cookie=72da4879e6311f52 in_seq=0 out_seq=0 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.101+0000 7f4379ffb640 1 -- 172.21.15.150:0/2343115299 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55acaad110c0 con 0x55acaad74ac0 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.101+0000 7f4379ffb640 10 client.25276 handle_command_reply: tid=0 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.101+0000 7f439b7b3740 1 client.25276 shutdown 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.101+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 msgr2=0x55acaad06820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.101+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4388015010 tx=0x7f43880046c0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 shutdown_connections 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55acaad74ac0 0x55acaad76ee0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acaacfe820 0x55acaacf8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4379ffb640 0 client.25276 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55acaacf5a50 0x55acaacf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 --2- 172.21.15.150:0/2343115299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55acaacf5130 0x55acaad06820 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 >> 172.21.15.150:0/2343115299 conn(0x55acaacefa50 msgr2=0x55acaacf1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 shutdown_connections 2024-03-18T16:00:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 1 -- 172.21.15.150:0/2343115299 wait complete. 2024-03-18T16:00:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f439b7b3740 20 client.25276 trim_cache size 0 max 16384 2024-03-18T16:00:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:00:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:00:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:00:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:00:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 msgr2=0x7f437c0a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 0x7f437c0a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f438c0eeb10 tx=0x7f438c02bef0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 msgr2=0x7f439419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 0x7f439419c8b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c430 tx=0x7f438c0336b0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 shutdown_connections 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f437c0a2b30 0x7f437c0a4ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43941077a0 0x7f439419c8b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4394102150 0x7f4394198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 --2- 172.21.15.150:0/3531935128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4394101790 0x7f4394197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 >> 172.21.15.150:0/3531935128 conn(0x7f43940fd2a0 msgr2=0x7f43940fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 shutdown_connections 2024-03-18T16:00:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:33.102+0000 7f4398bc6640 1 -- 172.21.15.150:0/3531935128 wait complete. 2024-03-18T16:00:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:33 smithi191 ceph-mon[36784]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:33 smithi156 ceph-mon[35919]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:33 smithi150 ceph-mon[31946]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-18T16:00:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:35 smithi191 ceph-mon[36784]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T16:00:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:35 smithi156 ceph-mon[35919]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T16:00:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:35 smithi150 ceph-mon[31946]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-18T16:00:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:37 smithi191 ceph-mon[36784]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-18T16:00:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:37 smithi156 ceph-mon[35919]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-18T16:00:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:37 smithi150 ceph-mon[31946]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-18T16:00:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:39 smithi191 ceph-mon[36784]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:39 smithi156 ceph-mon[35919]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:39 smithi150 ceph-mon[31946]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:41 smithi191 ceph-mon[36784]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:41 smithi156 ceph-mon[35919]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:41 smithi150 ceph-mon[31946]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-18T16:00:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:43 smithi191 ceph-mon[36784]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:43 smithi156 ceph-mon[35919]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:43 smithi150 ceph-mon[31946]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:45 smithi191 ceph-mon[36784]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-18T16:00:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:45 smithi156 ceph-mon[35919]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-18T16:00:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:45 smithi150 ceph-mon[31946]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 -- 172.21.15.150:0/1841132501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68281077c0 msgr2=0x7f6828107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 --2- 172.21.15.150:0/1841132501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68281077c0 0x7f6828107ba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6824009930 tx=0x7f682402bc30 comp rx=0 tx=0).stop 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 -- 172.21.15.150:0/1841132501 shutdown_connections 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 --2- 172.21.15.150:0/1841132501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6828102170 0x7f682810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 --2- 172.21.15.150:0/1841132501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 0x7f6828101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 --2- 172.21.15.150:0/1841132501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68281077c0 0x7f6828107ba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 -- 172.21.15.150:0/1841132501 >> 172.21.15.150:0/1841132501 conn(0x7f68280fd2e0 msgr2=0x7f68280ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 -- 172.21.15.150:0/1841132501 shutdown_connections 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.192+0000 7f682f5f4640 1 -- 172.21.15.150:0/1841132501 wait complete. 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 Processor -- start 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 -- start start 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 0x7f6828198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68281077c0 0x7f682819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6828111b90 con 0x7f6828102170 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6828111a10 con 0x7f68281077c0 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682f5f4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6828111d10 con 0x7f68281017b0 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54318/0 (socket says 172.21.15.150:54318) 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 -- 172.21.15.150:0/2240636818 learned_addr learned my addr 172.21.15.150:0/2240636818 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:46.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682d390640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 0x7f6828198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682db91640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68281077c0 0x7f682819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 -- 172.21.15.150:0/2240636818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 msgr2=0x7f6828198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 0x7f6828198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 -- 172.21.15.150:0/2240636818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68281077c0 msgr2=0x7f682819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68281077c0 0x7f682819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 -- 172.21.15.150:0/2240636818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682819d030 con 0x7f6828102170 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.193+0000 7f682cb8f640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f681800e9f0 tx=0x7f681800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.194+0000 7f681e7fc640 1 -- 172.21.15.150:0/2240636818 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681800cd80 con 0x7f6828102170 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.194+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682819d320 con 0x7f6828102170 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.194+0000 7f681e7fc640 1 -- 172.21.15.150:0/2240636818 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6818010910 con 0x7f6828102170 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.194+0000 7f681e7fc640 1 -- 172.21.15.150:0/2240636818 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68180188f0 con 0x7f6828102170 2024-03-18T16:00:46.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.194+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68281a4bb0 con 0x7f6828102170 2024-03-18T16:00:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.196+0000 7f681e7fc640 1 -- 172.21.15.150:0/2240636818 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 23) ==== 137467+0+0 (secure 0 0 0) 0x7f6818010ac0 con 0x7f6828102170 2024-03-18T16:00:46.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 Processor -- start 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f681e7fc640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 0x7f68140a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 -- start start 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f681e7fc640 1 -- 172.21.15.150:0/2240636818 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68180cf330 con 0x7f6828102170 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b5dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55baa4ab6a40 con 0x55baa4b52130 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55baa4a5f920 con 0x55baa4b52a50 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f68321e7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55baa4ab2ae0 con 0x55baa4b5b820 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682db91640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 0x7f68140a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54320/0 (socket says 172.21.15.150:54320) 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 -- 172.21.15.150:0/2962443232 learned_addr learned my addr 172.21.15.150:0/2962443232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682cb8f640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682db91640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b5dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 -- 172.21.15.150:0/2962443232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 msgr2=0x55baa4b5dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b5dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 -- 172.21.15.150:0/2962443232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 msgr2=0x55baa4b5b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.197+0000 7f682d390640 1 -- 172.21.15.150:0/2962443232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baa4a77460 con 0x55baa4b52130 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f682cb8f640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f682db91640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b5dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f682db91640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 0x7f68140a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6820005fd0 tx=0x7f6820011b40 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f682d390640 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f682402cfd0 tx=0x7f68240095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a92496e89de28f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f68137fe640 1 -- 172.21.15.150:0/2962443232 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824042070 con 0x55baa4b52130 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f68137fe640 1 -- 172.21.15.150:0/2962443232 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6824004030 con 0x55baa4b52130 2024-03-18T16:00:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.198+0000 7f68137fe640 1 -- 172.21.15.150:0/2962443232 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824004b80 con 0x55baa4b52130 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- 172.21.15.150:0/2962443232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 msgr2=0x55baa4b52510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f682402cfd0 tx=0x7f68240095f0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- 172.21.15.150:0/2962443232 shutdown_connections 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b5dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 --2- 172.21.15.150:0/2962443232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b52510 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- 172.21.15.150:0/2962443232 >> 172.21.15.150:0/2962443232 conn(0x55baa4b4ca50 msgr2=0x55baa4b4ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- 172.21.15.150:0/2962443232 shutdown_connections 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- 172.21.15.150:0/2962443232 wait complete. 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 -- start start 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.199+0000 7f68321e7740 1 Processor -- start 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b52130 0x55baa4b58860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b5b820 0x55baa4b592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55baa4b5e3f0 con 0x55baa4b52a50 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55baa4a72d70 con 0x55baa4b5b820 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55baa4b5e570 con 0x55baa4b52130 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f68321e7740 10 client.0 _ll_register_callbacks cb 0x55baa47caa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54332/0 (socket says 172.21.15.150:54332) 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 -- 172.21.15.150:0/4238375188 learned_addr learned my addr 172.21.15.150:0/4238375188 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682d390640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b52130 0x55baa4b58860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682db91640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b5b820 0x55baa4b592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 -- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b52130 msgr2=0x55baa4b58860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b52130 0x55baa4b58860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 -- 172.21.15.150:0/4238375188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b5b820 msgr2=0x55baa4b592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b5b820 0x55baa4b592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 -- 172.21.15.150:0/4238375188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baa47cb210 con 0x55baa4b52a50 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f682cb8f640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f681800e9c0 tx=0x7f681809ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eafb78109b41a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f6810ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.200+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818084e60 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68180a3d80 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6824004030 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f681809cba0 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 10 client.15462 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55baa4897620 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6824004030 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6818097670 con 0x55baa4b52a50 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 10 client.15462 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 20 client.15462 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4b6ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 4 client.15462 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:00:46.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baa4ac3f80 con 0x55baa4b6b9d0 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.201+0000 7f682d390640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4b6ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f682d390640 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4b6ddc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef4b0697901335e2 server_cookie=1b831d118bc6b96e in_seq=0 out_seq=0 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f6810ff9640 10 client.15462 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f6810ff9640 1 -- 172.21.15.150:0/4238375188 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55baa4ac3f80 con 0x55baa4b6b9d0 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f6810ff9640 10 client.15462 handle_command_reply: tid=0 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 client.15462 shutdown 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 msgr2=0x55baa4b58da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f681800e9c0 tx=0x7f681809ba60 comp rx=0 tx=0).stop 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 shutdown_connections 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4b6ddc0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b5b820 0x55baa4b592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52a50 0x55baa4b58da0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f6810ff9640 0 client.15462 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:46.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 --2- 172.21.15.150:0/4238375188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b52130 0x55baa4b58860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.202+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 >> 172.21.15.150:0/4238375188 conn(0x55baa4b4ca50 msgr2=0x55baa4b4e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.203+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 shutdown_connections 2024-03-18T16:00:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.203+0000 7f68321e7740 1 -- 172.21.15.150:0/4238375188 wait complete. 2024-03-18T16:00:46.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.203+0000 7f68321e7740 20 client.15462 trim_cache size 0 max 16384 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 Processor -- start 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 -- start start 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ada30 con 0x55baa4b52130 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68281ab300 con 0x55baa4b52a50 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f68321e7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f68281a9150 con 0x55baa4b5b820 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54338/0 (socket says 172.21.15.150:54338) 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f682d390640 1 -- 172.21.15.150:0/382455445 learned_addr learned my addr 172.21.15.150:0/382455445 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f682cb8f640 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.217+0000 7f682db91640 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 -- 172.21.15.150:0/382455445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 msgr2=0x55baa4b736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 -- 172.21.15.150:0/382455445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 msgr2=0x55baa4b67190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b67190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 -- 172.21.15.150:0/382455445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baa4b5dc10 con 0x55baa4b52130 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f682d390640 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f682402f020 tx=0x7f6824005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce9d187c1d0f1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f6812ffd640 1 -- 172.21.15.150:0/382455445 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824054070 con 0x55baa4b52130 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f6812ffd640 1 -- 172.21.15.150:0/382455445 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f682404f440 con 0x55baa4b52130 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.218+0000 7f6812ffd640 1 -- 172.21.15.150:0/382455445 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824042040 con 0x55baa4b52130 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- 172.21.15.150:0/382455445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 msgr2=0x55baa4b5af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f682402f020 tx=0x7f6824005e50 comp rx=0 tx=0).stop 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- 172.21.15.150:0/382455445 shutdown_connections 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4b736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b67190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- 172.21.15.150:0/382455445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5af90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- 172.21.15.150:0/382455445 >> 172.21.15.150:0/382455445 conn(0x55baa4b4ca50 msgr2=0x55baa4b47a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- 172.21.15.150:0/382455445 shutdown_connections 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- 172.21.15.150:0/382455445 wait complete. 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- start start 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 Processor -- start 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4bd0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68281a97c0 con 0x55baa4b52130 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f68281aac30 con 0x55baa4b52a50 2024-03-18T16:00:46.225 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f68281a9e90 con 0x55baa4b5b820 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f68321e7740 10 client.0 _ll_register_callbacks cb 0x7f6818097540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.219+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682d390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54340/0 (socket says 172.21.15.150:54340) 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682d390640 1 -- 172.21.15.150:0/500920249 learned_addr learned my addr 172.21.15.150:0/500920249 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682db91640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4bd0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 -- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 msgr2=0x55baa4bd0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4bd0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 -- 172.21.15.150:0/500920249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 msgr2=0x55baa4b5f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 -- 172.21.15.150:0/500920249 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baa4bd1220 con 0x55baa4b52a50 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682d390640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:46.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f682cb8f640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5fcc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68180cd050 tx=0x7f681801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df394629d5793b9a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f6810ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818084e60 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f682404f440 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f68180a3d80 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f681809ab60 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f68321e7740 10 client.25244 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.220+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55baa4bd24d0 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f682404f440 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f6818097040 con 0x55baa4b52a50 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 10 client.25244 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 10 client.25244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 20 client.25244 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4bd4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 4 client.25244 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:00:46.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55baa4abe2d0 con 0x55baa4b6b9d0 2024-03-18T16:00:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f682d390640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4bd4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f6810ff9640 10 client.25244 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:46.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.221+0000 7f682d390640 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4bd4c00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c95a13ab4c3e628 server_cookie=6538ca99521ada3 in_seq=0 out_seq=0 2024-03-18T16:00:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.227+0000 7f6810ff9640 1 -- 172.21.15.150:0/500920249 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55baa4abe2d0 con 0x55baa4b6b9d0 2024-03-18T16:00:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.227+0000 7f6810ff9640 10 client.25244 handle_command_reply: tid=0 2024-03-18T16:00:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 client.25244 shutdown 2024-03-18T16:00:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 msgr2=0x55baa4b5fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5fcc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68180cd050 tx=0x7f681801f000 comp rx=0 tx=0).stop 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 shutdown_connections 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55baa4b6b9d0 0x55baa4bd4c00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55baa4b5b820 0x55baa4bd0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4b52a50 0x55baa4b5fcc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 --2- 172.21.15.150:0/500920249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55baa4b52130 0x55baa4b5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f6810ff9640 0 client.25244 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 >> 172.21.15.150:0/500920249 conn(0x55baa4b4ca50 msgr2=0x55baa4b4dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 shutdown_connections 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 1 -- 172.21.15.150:0/500920249 wait complete. 2024-03-18T16:00:46.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f68321e7740 20 client.25244 trim_cache size 0 max 16384 2024-03-18T16:00:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:00:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:00:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:00:46.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 msgr2=0x7f68140a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 0x7f68140a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6820005fd0 tx=0x7f6820011b40 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 msgr2=0x7f68281985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.228+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f681800e9f0 tx=0x7f681800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 shutdown_connections 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f68140a2ab0 0x7f68140a4f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68281077c0 0x7f682819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6828102170 0x7f68281985d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 --2- 172.21.15.150:0/2240636818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68281017b0 0x7f6828198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 >> 172.21.15.150:0/2240636818 conn(0x7f68280fd2e0 msgr2=0x7f68280fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 shutdown_connections 2024-03-18T16:00:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:46.229+0000 7f682f5f4640 1 -- 172.21.15.150:0/2240636818 wait complete. 2024-03-18T16:00:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:47 smithi191 ceph-mon[36784]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:47 smithi156 ceph-mon[35919]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:47 smithi150 ceph-mon[31946]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-18T16:00:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:49 smithi191 ceph-mon[36784]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T16:00:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:49 smithi156 ceph-mon[35919]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T16:00:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:49 smithi150 ceph-mon[31946]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-18T16:00:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:51 smithi191 ceph-mon[36784]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T16:00:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:51 smithi156 ceph-mon[35919]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T16:00:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:51 smithi150 ceph-mon[31946]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-03-18T16:00:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:53 smithi191 ceph-mon[36784]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-18T16:00:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:53 smithi156 ceph-mon[35919]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-18T16:00:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:53 smithi150 ceph-mon[31946]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-18T16:00:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:55 smithi191 ceph-mon[36784]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:00:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:55 smithi156 ceph-mon[35919]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:00:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:55 smithi150 ceph-mon[31946]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:00:56.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:56 smithi191 ceph-mon[36784]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-18T16:00:57.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:56 smithi156 ceph-mon[35919]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-18T16:00:57.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:56 smithi150 ceph-mon[31946]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 -- 172.21.15.150:0/3296327838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5240106a10 msgr2=0x7f524010e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 --2- 172.21.15.150:0/3296327838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5240106a10 0x7f524010e6b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5238009ca0 tx=0x7f523802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 -- 172.21.15.150:0/3296327838 shutdown_connections 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 --2- 172.21.15.150:0/3296327838 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5240106a10 0x7f524010e6b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 --2- 172.21.15.150:0/3296327838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52401008a0 0x7f5240106330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 --2- 172.21.15.150:0/3296327838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52400ffef0 0x7f52401002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 -- 172.21.15.150:0/3296327838 >> 172.21.15.150:0/3296327838 conn(0x7f5240076c00 msgr2=0x7f52400fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 -- 172.21.15.150:0/3296327838 shutdown_connections 2024-03-18T16:00:59.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.361+0000 7f5244a73640 1 -- 172.21.15.150:0/3296327838 wait complete. 2024-03-18T16:00:59.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.362+0000 7f5244a73640 1 Processor -- start 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.362+0000 7f5244a73640 1 -- start start 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5240106a10 0x7f52401013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52401018e0 0x7f52401a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:53870/0 (socket says 172.21.15.150:53870) 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 -- 172.21.15.150:0/3347530504 learned_addr learned my addr 172.21.15.150:0/3347530504 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5240111b90 con 0x7f5240106a10 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5240111a10 con 0x7f52401018e0 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5240111d10 con 0x7f52400ffef0 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f5237fff640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5240106a10 0x7f52401013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52401018e0 0x7f52401a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 -- 172.21.15.150:0/3347530504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52401018e0 msgr2=0x7f52401a6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52401018e0 0x7f52401a6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 -- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5240106a10 msgr2=0x7f52401013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5240106a10 0x7f52401013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.363+0000 7f523e59c640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52401a6900 con 0x7f52400ffef0 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.364+0000 7f523e59c640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9f0 tx=0x7f523000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.364+0000 7f52377fe640 1 -- 172.21.15.150:0/3347530504 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000cd80 con 0x7f52400ffef0 2024-03-18T16:00:59.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.364+0000 7f52377fe640 1 -- 172.21.15.150:0/3347530504 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5230005500 con 0x7f52400ffef0 2024-03-18T16:00:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.365+0000 7f52377fe640 1 -- 172.21.15.150:0/3347530504 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230010690 con 0x7f52400ffef0 2024-03-18T16:00:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.365+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52401a6bf0 con 0x7f52400ffef0 2024-03-18T16:00:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.365+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401a70e0 con 0x7f52400ffef0 2024-03-18T16:00:59.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.368+0000 7f52377fe640 1 -- 172.21.15.150:0/3347530504 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 23) ==== 137467+0+0 (secure 0 0 0) 0x7f52300046c0 con 0x7f52400ffef0 2024-03-18T16:00:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 Processor -- start 2024-03-18T16:00:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 -- start start 2024-03-18T16:00:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f52377fe640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 0x7f522c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 0x5559e796f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 0x5559e7971c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5559e78caa40 con 0x5559e796f820 2024-03-18T16:00:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559e7873920 con 0x5559e7966a50 2024-03-18T16:00:59.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.369+0000 7f524765e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5559e78c6ae0 con 0x5559e7966130 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f5237fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 0x5559e796f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 0x5559e7971c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:53878/0 (socket says 172.21.15.150:53878) 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 -- 172.21.15.150:0/3386302140 learned_addr learned my addr 172.21.15.150:0/3386302140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 -- 172.21.15.150:0/3386302140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 msgr2=0x5559e796f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 0x5559e796f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 -- 172.21.15.150:0/3386302140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 msgr2=0x5559e7971c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 0x5559e7971c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f52377fe640 1 -- 172.21.15.150:0/3347530504 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f52300cf3c0 con 0x7f52400ffef0 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 -- 172.21.15.150:0/3386302140 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559e788b460 con 0x5559e7966130 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f5237fff640 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 0x5559e796f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 0x5559e7971c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 0x7f522c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f523e59c640 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f523009b5a0 tx=0x7f5230004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a88ac662bba47a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.370+0000 7f5234ff9640 1 -- 172.21.15.150:0/3386302140 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523009bfa0 con 0x5559e7966130 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.371+0000 7f5234ff9640 1 -- 172.21.15.150:0/3386302140 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523009b8a0 con 0x5559e7966130 2024-03-18T16:00:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.371+0000 7f5234ff9640 1 -- 172.21.15.150:0/3386302140 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230097f10 con 0x5559e7966130 2024-03-18T16:00:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- 172.21.15.150:0/3386302140 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 msgr2=0x5559e7966510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 0x7f522c0a50a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5238004af0 tx=0x7f5238005c50 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f523009b5a0 tx=0x7f5230004a40 comp rx=0 tx=0).stop 2024-03-18T16:00:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- 172.21.15.150:0/3386302140 shutdown_connections 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e796f820 0x5559e7971c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966a50 0x5559e796f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 --2- 172.21.15.150:0/3386302140 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966130 0x5559e7966510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- 172.21.15.150:0/3386302140 >> 172.21.15.150:0/3386302140 conn(0x5559e7960a50 msgr2=0x5559e7960e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- 172.21.15.150:0/3386302140 shutdown_connections 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- 172.21.15.150:0/3386302140 wait complete. 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 -- start start 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.372+0000 7f524765e740 1 Processor -- start 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966a50 0x5559e796ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e796f820 0x5559e796d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5559e79723f0 con 0x5559e7966130 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559e7886d70 con 0x5559e796f820 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5559e7972570 con 0x5559e7966a50 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f524765e740 10 client.0 _ll_register_callbacks cb 0x5559e75dea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e796f820 0x5559e796d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36106/0 (socket says 172.21.15.150:36106) 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 -- 172.21.15.150:0/548023361 learned_addr learned my addr 172.21.15.150:0/548023361 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f5237fff640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966a50 0x5559e796ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 -- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966a50 msgr2=0x5559e796ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966a50 0x5559e796ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 -- 172.21.15.150:0/548023361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e796f820 msgr2=0x5559e796d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e796f820 0x5559e796d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.373+0000 7f523e59c640 1 -- 172.21.15.150:0/548023361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559e75df210 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f523e59c640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5230004c00 tx=0x7f52300ab360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398a42bf6220082d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f5225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300cc980 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f52300860d0 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f523009b8a0 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f52300b3030 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f524765e740 10 client.15480 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5559e76ab620 con 0x5559e7966130 2024-03-18T16:00:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.374+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523009b8a0 con 0x5559e7966130 2024-03-18T16:00:59.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f523009ccd0 con 0x5559e7966130 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 10 client.15480 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 20 client.15480 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e7981c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 4 client.15480 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559e78d7f80 con 0x5559e797f860 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.375+0000 7f5237fff640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e7981c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.376+0000 7f5237fff640 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e7981c50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8b16a206c9abda8 server_cookie=51a231951842ee67 in_seq=0 out_seq=0 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.376+0000 7f5225ffb640 10 client.15480 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.376+0000 7f5225ffb640 1 -- 172.21.15.150:0/548023361 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559e78d7f80 con 0x5559e797f860 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.376+0000 7f5225ffb640 10 client.15480 handle_command_reply: tid=0 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.376+0000 7f524765e740 1 client.15480 shutdown 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 msgr2=0x5559e796c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5230004c00 tx=0x7f52300ab360 comp rx=0 tx=0).stop 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 shutdown_connections 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e7981c50 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e796f820 0x5559e796d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f5225ffb640 0 client.15480 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e7966a50 0x5559e796ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 --2- 172.21.15.150:0/548023361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966130 0x5559e796c8f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 >> 172.21.15.150:0/548023361 conn(0x5559e7960a50 msgr2=0x5559e79627a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 shutdown_connections 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 1 -- 172.21.15.150:0/548023361 wait complete. 2024-03-18T16:00:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.377+0000 7f524765e740 20 client.15480 trim_cache size 0 max 16384 2024-03-18T16:00:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 Processor -- start 2024-03-18T16:00:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 -- start start 2024-03-18T16:00:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e7984c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e7969b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f52401b1ce0 con 0x5559e7966a50 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f52401af5b0 con 0x5559e7966130 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.406+0000 7f524765e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ad4b0 con 0x5559e796f820 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f5237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e7984c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:43702/0 (socket says 172.21.15.150:43702) 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 -- 172.21.15.150:0/3656642358 learned_addr learned my addr 172.21.15.150:0/3656642358 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e7969b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 -- 172.21.15.150:0/3656642358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 msgr2=0x5559e7969b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e7969b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 -- 172.21.15.150:0/3656642358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 msgr2=0x5559e7984c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e7984c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 -- 172.21.15.150:0/3656642358 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559e7971f80 con 0x5559e7966130 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523ed9d640 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e7969b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f523e59c640 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f523009b830 tx=0x7f523009b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf866cfb26b8d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.407+0000 7f5227fff640 1 -- 172.21.15.150:0/3656642358 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523009b200 con 0x5559e7966130 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.408+0000 7f5227fff640 1 -- 172.21.15.150:0/3656642358 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f523009bfa0 con 0x5559e7966130 2024-03-18T16:00:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.408+0000 7f5227fff640 1 -- 172.21.15.150:0/3656642358 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523009ce30 con 0x5559e7966130 2024-03-18T16:00:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.408+0000 7f524765e740 1 -- 172.21.15.150:0/3656642358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 msgr2=0x5559e796f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.408+0000 7f524765e740 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f523009b830 tx=0x7f523009b670 comp rx=0 tx=0).stop 2024-03-18T16:00:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 -- 172.21.15.150:0/3656642358 shutdown_connections 2024-03-18T16:00:59.416 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e7969b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e7984c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 --2- 172.21.15.150:0/3656642358 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e796f020 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 -- 172.21.15.150:0/3656642358 >> 172.21.15.150:0/3656642358 conn(0x5559e7960a50 msgr2=0x5559e795fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 -- 172.21.15.150:0/3656642358 shutdown_connections 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 -- 172.21.15.150:0/3656642358 wait complete. 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 -- start start 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.409+0000 7f524765e740 1 Processor -- start 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e79de840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ada70 con 0x5559e7966a50 2024-03-18T16:00:59.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f523e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36128/0 (socket says 172.21.15.150:36128) 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f523e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:43708/0 (socket says 172.21.15.150:43708) 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 -- 172.21.15.150:0/51010164 learned_addr learned my addr 172.21.15.150:0/51010164 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f523ed9d640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e79de840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f52401aeee0 con 0x5559e7966130 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f52401ae140 con 0x5559e796f820 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 10 client.0 _ll_register_callbacks cb 0x7f5228005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 -- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 msgr2=0x5559e79de840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e79de840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 -- 172.21.15.150:0/51010164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 msgr2=0x5559e7982ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f5237fff640 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559e79dee00 con 0x5559e7966a50 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f523e59c640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:00:59.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.410+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5559e78d22d0 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5237fff640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52401028f0 tx=0x7f5228020710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b70e6f7d500357 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5225ffb640 10 client.15486 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5225ffb640 1 -- 172.21.15.150:0/51010164 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228003a00 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5225ffb640 1 -- 172.21.15.150:0/51010164 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f5228003ba0 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5225ffb640 1 -- 172.21.15.150:0/51010164 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5228003e00 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f524765e740 10 client.15486 fetch_fsmap learned FSMap version 22 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({fsmap=22}) -- 0x5559e79e6130 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.411+0000 7f5225ffb640 1 -- 172.21.15.150:0/51010164 <== mon.0 v2:172.21.15.150:3300/0 4 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5228010430 con 0x5559e7966a50 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 10 client.15486 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 10 client.15486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 20 client.15486 populate_metadata read hostname 'smithi150' 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e79e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 4 client.15486 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5559e7850770 con 0x5559e797f860 2024-03-18T16:00:59.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f523e59c640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e79e87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:59.420 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f5225ffb640 10 client.15486 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:59.420 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.412+0000 7f523e59c640 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e79e87b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=405cdd051ff2bb10 server_cookie=3c6d464e62913f79 in_seq=0 out_seq=0 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.415+0000 7f5225ffb640 1 -- 172.21.15.150:0/51010164 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5559e7850770 con 0x5559e797f860 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.415+0000 7f5225ffb640 10 client.15486 handle_command_reply: tid=0 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.415+0000 7f524765e740 1 client.15486 shutdown 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 msgr2=0x5559e79833e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52401028f0 tx=0x7f5228020710 comp rx=0 tx=0).stop 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 shutdown_connections 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5559e797f860 0x5559e79e87b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5559e796f820 0x5559e79de840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5559e7966a50 0x5559e79833e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f5225ffb640 0 client.15486 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 --2- 172.21.15.150:0/51010164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559e7966130 0x5559e7982ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 >> 172.21.15.150:0/51010164 conn(0x5559e7960a50 msgr2=0x5559e797a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 shutdown_connections 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 1 -- 172.21.15.150:0/51010164 wait complete. 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.416+0000 7f524765e740 20 client.15486 trim_cache size 0 max 16384 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:00:59.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:00:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 msgr2=0x7f522c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 0x7f522c0a50a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5238004af0 tx=0x7f5238005c50 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 msgr2=0x7f5240100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9f0 tx=0x7f523000edf0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 shutdown_connections 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.417+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f522c0a2be0 0x7f522c0a50a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52401018e0 0x7f52401a6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5240106a10 0x7f52401013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 --2- 172.21.15.150:0/3347530504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52400ffef0 0x7f5240100e60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 >> 172.21.15.150:0/3347530504 conn(0x7f5240076c00 msgr2=0x7f5240106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 shutdown_connections 2024-03-18T16:00:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:00:59.418+0000 7f5244a73640 1 -- 172.21.15.150:0/3347530504 wait complete. 2024-03-18T16:00:59.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:00:59 smithi150 ceph-mon[31946]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:00:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:00:59 smithi191 ceph-mon[36784]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:01:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:00:59 smithi156 ceph-mon[35919]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-03-18T16:01:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:01:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:01:00.960+0000 7f91fa0d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:01:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:01:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:01:00.960+0000 7f91fa0d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:01:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:01:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:01:00.961+0000 7f91fa0d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:01:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:01:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:01:00.961+0000 7f91fa0d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:01:01.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:01:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:01:00.961+0000 7f91fa0d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:01:01.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:01 smithi191 ceph-mon[36784]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1023 B/s wr, 9 op/s 2024-03-18T16:01:01.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:01 smithi156 ceph-mon[35919]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1023 B/s wr, 9 op/s 2024-03-18T16:01:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:01 smithi150 ceph-mon[31946]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1023 B/s wr, 9 op/s 2024-03-18T16:01:03.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:03 smithi150 ceph-mon[31946]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T16:01:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:03 smithi191 ceph-mon[36784]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T16:01:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:03 smithi156 ceph-mon[35919]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-18T16:01:04.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:04 smithi150 ceph-mon[31946]: mgrmap e24: x(active, since 59s), standbys: y, z 2024-03-18T16:01:04.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:04 smithi191 ceph-mon[36784]: mgrmap e24: x(active, since 59s), standbys: y, z 2024-03-18T16:01:05.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:04 smithi156 ceph-mon[35919]: mgrmap e24: x(active, since 59s), standbys: y, z 2024-03-18T16:01:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:05 smithi150 ceph-mon[31946]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T16:01:05.863 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:05 smithi191 ceph-mon[36784]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T16:01:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:05 smithi156 ceph-mon[35919]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-18T16:01:07.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:07 smithi150 ceph-mon[31946]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:07 smithi191 ceph-mon[36784]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:07 smithi156 ceph-mon[35919]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:09.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:09 smithi150 ceph-mon[31946]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-18T16:01:09.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:09 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:09 smithi191 ceph-mon[36784]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-18T16:01:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:09 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:09 smithi156 ceph-mon[35919]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-18T16:01:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:09 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:11.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:11.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:10 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:11.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:11.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:11.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:10 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-18T16:01:11.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:11.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:11.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:10 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:01:12.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- 172.21.15.150:0/1521616495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac107840 msgr2=0x7f8bac107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 --2- 172.21.15.150:0/1521616495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac107840 0x7f8bac107c20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009890 tx=0x7f8b9c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- 172.21.15.150:0/1521616495 shutdown_connections 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 --2- 172.21.15.150:0/1521616495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac069800 0x7f8bac10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 --2- 172.21.15.150:0/1521616495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac068e60 0x7f8bac0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 --2- 172.21.15.150:0/1521616495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac107840 0x7f8bac107c20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- 172.21.15.150:0/1521616495 >> 172.21.15.150:0/1521616495 conn(0x7f8bac077530 msgr2=0x7f8bac077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- 172.21.15.150:0/1521616495 shutdown_connections 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- 172.21.15.150:0/1521616495 wait complete. 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 Processor -- start 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.513+0000 7f8bb2910640 1 -- start start 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac069800 0x7f8bac198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111ba0 con 0x7f8bac107840 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111a20 con 0x7f8bac069800 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb2910640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac111d20 con 0x7f8bac068e60 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33284/0 (socket says 172.21.15.150:33284) 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 -- 172.21.15.150:0/442696988 learned_addr learned my addr 172.21.15.150:0/442696988 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38948/0 (socket says 172.21.15.150:38948) 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac069800 0x7f8bac198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 -- 172.21.15.150:0/442696988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac069800 msgr2=0x7f8bac198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac069800 0x7f8bac198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 -- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 msgr2=0x7f8bac19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 -- 172.21.15.150:0/442696988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac19cfe0 con 0x7f8bac068e60 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8bb0ead640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.514+0000 7f8babfff640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c000c00 tx=0x7f8b9c004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.515+0000 7f8ba97fa640 1 -- 172.21.15.150:0/442696988 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c041020 con 0x7f8bac068e60 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.515+0000 7f8ba97fa640 1 -- 172.21.15.150:0/442696988 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c02c9f0 con 0x7f8bac068e60 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.515+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac19d270 con 0x7f8bac068e60 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.515+0000 7f8ba97fa640 1 -- 172.21.15.150:0/442696988 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c039040 con 0x7f8bac068e60 2024-03-18T16:01:12.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.515+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac1a4b60 con 0x7f8bac068e60 2024-03-18T16:01:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8ba97fa640 1 -- 172.21.15.150:0/442696988 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 24) ==== 137524+0+0 (secure 0 0 0) 0x7f8b9c044020 con 0x7f8bac068e60 2024-03-18T16:01:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8ba97fa640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 0x7f8b940a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8bb55de740 1 Processor -- start 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8bb55de740 1 -- start start 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 0x7f8b940a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.517+0000 7f8ba97fa640 1 -- 172.21.15.150:0/442696988 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8b9c0f07c0 con 0x7f8bac068e60 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6857dc30 0x555d68580020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555d684d4a40 con 0x555d685704f0 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555d6847d920 con 0x555d6857dc30 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb55de740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555d684d0ae0 con 0x555d6856fbd0 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33290/0 (socket says 172.21.15.150:33290) 2024-03-18T16:01:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 -- 172.21.15.150:0/1030252941 learned_addr learned my addr 172.21.15.150:0/1030252941 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bb0ead640 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6857dc30 0x555d68580020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 0x7f8b940a4ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bac077aa0 tx=0x7f8ba0011440 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 -- 172.21.15.150:0/1030252941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6857dc30 msgr2=0x555d68580020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6857dc30 0x555d68580020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 -- 172.21.15.150:0/1030252941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 msgr2=0x555d6857d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 -- 172.21.15.150:0/1030252941 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d68495460 con 0x555d6856fbd0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8babfff640 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b9190 tx=0x7f8b9c02bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba05c116b77b61a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.518+0000 7f8b927fc640 1 -- 172.21.15.150:0/1030252941 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0bcd30 con 0x555d6856fbd0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8b927fc640 1 -- 172.21.15.150:0/1030252941 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c0c0e80 con 0x555d6856fbd0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8b927fc640 1 -- 172.21.15.150:0/1030252941 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0c5040 con 0x555d6856fbd0 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 -- 172.21.15.150:0/1030252941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 msgr2=0x555d6856ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b9190 tx=0x7f8b9c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 -- 172.21.15.150:0/1030252941 shutdown_connections 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6857dc30 0x555d68580020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 --2- 172.21.15.150:0/1030252941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d6856ffb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 -- 172.21.15.150:0/1030252941 >> 172.21.15.150:0/1030252941 conn(0x555d6856aa50 msgr2=0x555d6856ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 -- 172.21.15.150:0/1030252941 shutdown_connections 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.519+0000 7f8bb55de740 1 -- 172.21.15.150:0/1030252941 wait complete. 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 -- start start 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 Processor -- start 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 0x555d6857b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555d68580620 con 0x555d685704f0 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555d68490d70 con 0x555d6856fbd0 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555d685807a0 con 0x555d6857dc30 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb55de740 10 client.0 _ll_register_callbacks cb 0x555d681e8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38968/0 (socket says 172.21.15.150:38968) 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bab7fe640 1 -- 172.21.15.150:0/343317348 learned_addr learned my addr 172.21.15.150:0/343317348 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8babfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.520+0000 7f8bb0ead640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 0x555d6857b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 -- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 msgr2=0x555d6857b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 0x555d6857b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 -- 172.21.15.150:0/343317348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 msgr2=0x555d6857b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 -- 172.21.15.150:0/343317348 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d681e9210 con 0x555d6856fbd0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8bb0ead640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 0x555d6857b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8babfff640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c02cbc0 tx=0x7f8b9c0ede90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4d138cb34a3516 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0c54d0 con 0x555d6856fbd0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b9c0c0e80 con 0x555d6856fbd0 2024-03-18T16:01:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8b9c0c5040 con 0x555d6856fbd0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8b9c09f020 con 0x555d6856fbd0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8bb55de740 10 client.25277 fetch_fsmap learned FSMap version 22 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555d682b5620 con 0x555d6856fbd0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9c0c0e80 con 0x555d6856fbd0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.521+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8b9c03c030 con 0x555d6856fbd0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 10 client.25277 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 10 client.25277 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 20 client.25277 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d6858d970 0x555d6858fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 4 client.25277 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d684e1f80 con 0x555d6858d970 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d6858d970 0x555d6858fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d6858d970 0x555d6858fd60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c9a046214cc3fc server_cookie=22bf3a31e6119ff in_seq=0 out_seq=0 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.522+0000 7f8b8b7fe640 10 client.25277 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:12.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/343317348 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d684e1f80 con 0x555d6858d970 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8b8b7fe640 10 client.25277 handle_command_reply: tid=0 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 client.25277 shutdown 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 msgr2=0x555d6857ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c02cbc0 tx=0x7f8b9c0ede90 comp rx=0 tx=0).stop 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 shutdown_connections 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d6858d970 0x555d6858fd60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6857dc30 0x555d6857b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d685704f0 0x555d6857b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8b8b7fe640 0 client.25277 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.523+0000 7f8bb55de740 1 --2- 172.21.15.150:0/343317348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857ac20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.524+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 >> 172.21.15.150:0/343317348 conn(0x555d6856aa50 msgr2=0x555d6856c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.524+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 shutdown_connections 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.524+0000 7f8bb55de740 1 -- 172.21.15.150:0/343317348 wait complete. 2024-03-18T16:01:12.531 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.524+0000 7f8bb55de740 20 client.25277 trim_cache size 0 max 16384 2024-03-18T16:01:12.550 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 Processor -- start 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 -- start start 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68565a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1ad910 con 0x555d6857dc30 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1ab1e0 con 0x555d6856fbd0 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb55de740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a9030 con 0x555d685704f0 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68565a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33306/0 (socket says 172.21.15.150:33306) 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68565a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38970/0 (socket says 172.21.15.150:38970) 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8bab7fe640 1 -- 172.21.15.150:0/2183141571 learned_addr learned my addr 172.21.15.150:0/2183141571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.548+0000 7f8babfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.551 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 -- 172.21.15.150:0/2183141571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 msgr2=0x555d6857d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 -- 172.21.15.150:0/2183141571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 msgr2=0x555d68565a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68565a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 -- 172.21.15.150:0/2183141571 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d68579bf0 con 0x555d685704f0 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x555d68590540 tx=0x7f8ba001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63512a0bbdfbb7f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8b91ffb640 1 -- 172.21.15.150:0/2183141571 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0017040 con 0x555d685704f0 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8b91ffb640 1 -- 172.21.15.150:0/2183141571 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0019050 con 0x555d685704f0 2024-03-18T16:01:12.552 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.549+0000 7f8b91ffb640 1 -- 172.21.15.150:0/2183141571 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0005180 con 0x555d685704f0 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- 172.21.15.150:0/2183141571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 msgr2=0x555d685893c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x555d68590540 tx=0x7f8ba001c000 comp rx=0 tx=0).stop 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- 172.21.15.150:0/2183141571 shutdown_connections 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68565a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d685704f0 0x555d685893c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2183141571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d6856fbd0 0x555d6857d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- 172.21.15.150:0/2183141571 >> 172.21.15.150:0/2183141571 conn(0x555d6856aa50 msgr2=0x555d6856c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- 172.21.15.150:0/2183141571 shutdown_connections 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- 172.21.15.150:0/2183141571 wait complete. 2024-03-18T16:01:12.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.550+0000 7f8bb55de740 1 -- start start 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 Processor -- start 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 0x555d685829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a96a0 con 0x555d6857dc30 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1aab10 con 0x555d685704f0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a9d70 con 0x555d6856fbd0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb55de740 10 client.0 _ll_register_callbacks cb 0x7f8ba0009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb0ead640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38984/0 (socket says 172.21.15.150:38984) 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bab7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 0x555d685829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8bb0ead640 1 -- 172.21.15.150:0/2282842508 learned_addr learned my addr 172.21.15.150:0/2282842508 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:12.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8babfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33308/0 (socket says 172.21.15.150:33308) 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8babfff640 1 -- 172.21.15.150:0/2282842508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 msgr2=0x555d685829c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8babfff640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 0x555d685829c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.551+0000 7f8babfff640 1 -- 172.21.15.150:0/2282842508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 msgr2=0x555d68582f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8babfff640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8babfff640 1 -- 172.21.15.150:0/2282842508 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d685900d0 con 0x555d6856fbd0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8bb0ead640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 0x555d685829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8babfff640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0bafe0 tx=0x7f8b9c032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f8f51b3163df16b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0c54d0 con 0x555d6856fbd0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ba0019050 con 0x555d6856fbd0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8b9c0c5040 con 0x555d6856fbd0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8b9c0c0e80 con 0x555d6856fbd0 2024-03-18T16:01:12.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8bb55de740 10 client.25336 fetch_fsmap learned FSMap version 22 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555d685900d0 con 0x555d6856fbd0 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0019050 con 0x555d6856fbd0 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.552+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f8b9c09bb50 con 0x555d6856fbd0 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 10 client.25336 fetch_fsmap finished waiting for FSMap version 22 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 10 client.25336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 20 client.25336 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d685f49f0 0x555d685f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 4 client.25336 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d684dc2d0 con 0x555d685f49f0 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d685f49f0 0x555d685f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8b8b7fe640 10 client.25336 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:12.559 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.553+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d685f49f0 0x555d685f6de0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33de96a2656d6afa server_cookie=9a77f41288946ed6 in_seq=0 out_seq=0 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8b8b7fe640 1 -- 172.21.15.150:0/2282842508 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d684dc2d0 con 0x555d685f49f0 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8b8b7fe640 10 client.25336 handle_command_reply: tid=0 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 client.25336 shutdown 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 msgr2=0x555d685859f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0bafe0 tx=0x7f8b9c032f90 comp rx=0 tx=0).stop 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 shutdown_connections 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555d685f49f0 0x555d685f6de0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555d6857dc30 0x555d68582f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8b8b7fe640 0 client.25336 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d685704f0 0x555d685829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 --2- 172.21.15.150:0/2282842508 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d6856fbd0 0x555d685859f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.561 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 >> 172.21.15.150:0/2282842508 conn(0x555d6856aa50 msgr2=0x555d6856bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 shutdown_connections 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 1 -- 172.21.15.150:0/2282842508 wait complete. 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.558+0000 7f8bb55de740 20 client.25336 trim_cache size 0 max 16384 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:01:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 msgr2=0x7f8b940a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 0x7f8b940a4ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8bac077aa0 tx=0x7f8ba0011440 comp rx=0 tx=0).stop 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 msgr2=0x7f8bac198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c000c00 tx=0x7f8b9c004550 comp rx=0 tx=0).stop 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 shutdown_connections 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8b940a2b30 0x7f8b940a4ff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac107840 0x7f8bac19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bac069800 0x7f8bac198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 --2- 172.21.15.150:0/442696988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bac068e60 0x7f8bac198040 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 >> 172.21.15.150:0/442696988 conn(0x7f8bac077530 msgr2=0x7f8bac0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 shutdown_connections 2024-03-18T16:01:12.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:12.559+0000 7f8bb2910640 1 -- 172.21.15.150:0/442696988 wait complete. 2024-03-18T16:01:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:13 smithi150 ceph-mon[31946]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:13 smithi191 ceph-mon[36784]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:13 smithi156 ceph-mon[35919]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:Note: switching to '4e19f704afd9957ea657ab858f9ecf76401fef6f'. 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T16:01:14.093 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: git switch -c 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr:Or undo this operation with: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: git switch - 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-03-18T16:01:14.094 INFO:tasks.workunit.client.0.smithi150.stderr:HEAD is now at 4e19f704afd ignorelist mon down 2024-03-18T16:01:14.098 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T16:01:14.158 INFO:tasks.workunit.client.0.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T16:01:14.160 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T16:01:14.161 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T16:01:14.924 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T16:01:14.990 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T16:01:15.031 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T16:01:15.032 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T16:01:15.032 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T16:01:15.071 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T16:01:15.075 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T16:01:15.075 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T16:01:15.130 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-18T16:01:15.132 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-18T16:01:15.132 DEBUG:teuthology.orchestra.run.smithi150:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-18T16:01:15.192 INFO:tasks.workunit.client.0.smithi150.stderr:+ dbench 1 2024-03-18T16:01:15.203 INFO:tasks.workunit.client.0.smithi150.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-18T16:01:15.203 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-03-18T16:01:15.203 INFO:tasks.workunit.client.0.smithi150.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-18T16:01:15.211 INFO:tasks.workunit.client.0.smithi150.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-18T16:01:15.311 INFO:tasks.workunit.client.0.smithi150.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-18T16:01:15.312 INFO:tasks.workunit.client.0.smithi150.stdout:releasing clients 2024-03-18T16:01:15.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:15 smithi156 ceph-mon[35919]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:15.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:15 smithi150 ceph-mon[31946]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:15 smithi191 ceph-mon[36784]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-18T16:01:16.311 INFO:tasks.workunit.client.0.smithi150.stdout: 1 2926 33.48 MB/sec warmup 1 sec latency 8.204 ms 2024-03-18T16:01:17.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 5012 22.57 MB/sec warmup 2 sec latency 16.870 ms 2024-03-18T16:01:17.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:17 smithi150 ceph-mon[31946]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 26 op/s 2024-03-18T16:01:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:17 smithi191 ceph-mon[36784]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 26 op/s 2024-03-18T16:01:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:17 smithi156 ceph-mon[35919]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 33 KiB/s wr, 26 op/s 2024-03-18T16:01:18.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 6007 15.74 MB/sec warmup 3 sec latency 744.173 ms 2024-03-18T16:01:18.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:18 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:18 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:18 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 6007 11.81 MB/sec warmup 4 sec latency 1744.291 ms 2024-03-18T16:01:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:19 smithi150 ceph-mon[31946]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-18T16:01:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:19 smithi191 ceph-mon[36784]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-18T16:01:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:19 smithi156 ceph-mon[35919]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-18T16:01:20.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 6007 9.45 MB/sec warmup 5 sec latency 2744.394 ms 2024-03-18T16:01:21.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 6007 7.87 MB/sec warmup 6 sec latency 3744.456 ms 2024-03-18T16:01:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:21 smithi150 ceph-mon[31946]: pgmap v41: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 307 KiB/s wr, 43 op/s 2024-03-18T16:01:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:21 smithi191 ceph-mon[36784]: pgmap v41: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 307 KiB/s wr, 43 op/s 2024-03-18T16:01:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:21 smithi156 ceph-mon[35919]: pgmap v41: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 307 KiB/s wr, 43 op/s 2024-03-18T16:01:22.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 6007 6.75 MB/sec warmup 7 sec latency 4744.572 ms 2024-03-18T16:01:23.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 7768 7.41 MB/sec warmup 8 sec latency 4927.186 ms 2024-03-18T16:01:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:23 smithi150 ceph-mon[31946]: pgmap v42: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:23 smithi191 ceph-mon[36784]: pgmap v42: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:23 smithi156 ceph-mon[35919]: pgmap v42: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:24.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10719 8.54 MB/sec warmup 9 sec latency 92.390 ms 2024-03-18T16:01:25.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10719 7.68 MB/sec warmup 10 sec latency 1092.497 ms 2024-03-18T16:01:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 -- 172.21.15.150:0/1758667707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c107860 msgr2=0x7fa91c107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 --2- 172.21.15.150:0/1758667707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c107860 0x7fa91c107c40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009cd0 tx=0x7fa90c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 -- 172.21.15.150:0/1758667707 shutdown_connections 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 --2- 172.21.15.150:0/1758667707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c06a9a0 0x7fa91c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 --2- 172.21.15.150:0/1758667707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 0x7fa91c06a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 --2- 172.21.15.150:0/1758667707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c107860 0x7fa91c107c40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.656+0000 7fa9219a9640 1 -- 172.21.15.150:0/1758667707 >> 172.21.15.150:0/1758667707 conn(0x7fa91c0774b0 msgr2=0x7fa91c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.657+0000 7fa9219a9640 1 -- 172.21.15.150:0/1758667707 shutdown_connections 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.657+0000 7fa9219a9640 1 -- 172.21.15.150:0/1758667707 wait complete. 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.657+0000 7fa9219a9640 1 Processor -- start 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.657+0000 7fa9219a9640 1 -- start start 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 0x7fa91c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 0x7fa91c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c111bc0 con 0x7fa91c06a9a0 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c111a40 con 0x7fa91c069fe0 2024-03-18T16:01:25.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c111d40 con 0x7fa91c107860 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35802/0 (socket says 172.21.15.150:35802) 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 -- 172.21.15.150:0/3968771746 learned_addr learned my addr 172.21.15.150:0/3968771746 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 0x7fa91c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91affd640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 0x7fa91c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 -- 172.21.15.150:0/3968771746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 msgr2=0x7fa91c19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 0x7fa91c19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 -- 172.21.15.150:0/3968771746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 msgr2=0x7fa91c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 0x7fa91c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 -- 172.21.15.150:0/3968771746 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c19cf40 con 0x7fa91c06a9a0 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 0x7fa91c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa91a7fc640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa91000ea60 tx=0x7fa91000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9209a7640 1 -- 172.21.15.150:0/3968771746 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91000cde0 con 0x7fa91c06a9a0 2024-03-18T16:01:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9209a7640 1 -- 172.21.15.150:0/3968771746 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9100035f0 con 0x7fa91c06a9a0 2024-03-18T16:01:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c19d230 con 0x7fa91c06a9a0 2024-03-18T16:01:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9209a7640 1 -- 172.21.15.150:0/3968771746 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910010670 con 0x7fa91c06a9a0 2024-03-18T16:01:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.658+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a4ac0 con 0x7fa91c06a9a0 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa9209a7640 1 -- 172.21.15.150:0/3968771746 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) ==== 137524+0+0 (secure 0 0 0) 0x7fa9100037a0 con 0x7fa91c06a9a0 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 Processor -- start 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 -- start start 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x562650148510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa9209a7640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 0x7fa9040a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650153c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5626500aca40 con 0x562650148a50 2024-03-18T16:01:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562650055920 con 0x562650148130 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.661+0000 7fa92458d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5626500a8ae0 con 0x562650151820 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35812/0 (socket says 172.21.15.150:35812) 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x562650148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 -- 172.21.15.150:0/2806499200 learned_addr learned my addr 172.21.15.150:0/2806499200 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91a7fc640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 0x7fa9040a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa9209a7640 1 -- 172.21.15.150:0/3968771746 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa9100cf320 con 0x7fa91c06a9a0 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 -- 172.21.15.150:0/2806499200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 msgr2=0x562650153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 -- 172.21.15.150:0/2806499200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 msgr2=0x562650148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x562650148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 -- 172.21.15.150:0/2806499200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56265006d460 con 0x562650148a50 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x562650148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91a7fc640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 0x7fa9040a5070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9100cc9a0 tx=0x7fa91006e460 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa91affd640 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009ca0 tx=0x7fa90c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381f6dde4e4f5aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa901ffb640 1 -- 172.21.15.150:0/2806499200 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c043020 con 0x562650148a50 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa901ffb640 1 -- 172.21.15.150:0/2806499200 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c032c90 con 0x562650148a50 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.662+0000 7fa901ffb640 1 -- 172.21.15.150:0/2806499200 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039040 con 0x562650148a50 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- 172.21.15.150:0/2806499200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 msgr2=0x5626501512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009ca0 tx=0x7fa90c009880 comp rx=0 tx=0).stop 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- 172.21.15.150:0/2806499200 shutdown_connections 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x5626501512e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 --2- 172.21.15.150:0/2806499200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x562650148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- 172.21.15.150:0/2806499200 >> 172.21.15.150:0/2806499200 conn(0x562650142a50 msgr2=0x562650142e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- 172.21.15.150:0/2806499200 shutdown_connections 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- 172.21.15.150:0/2806499200 wait complete. 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.663+0000 7fa92458d740 1 -- start start 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 Processor -- start 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x56265014e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x56265014f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5626501543f0 con 0x562650148a50 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562650068d70 con 0x562650148130 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562650154570 con 0x562650151820 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa92458d740 10 client.0 _ll_register_callbacks cb 0x56264fdc0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35816/0 (socket says 172.21.15.150:35816) 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 -- 172.21.15.150:0/127103526 learned_addr learned my addr 172.21.15.150:0/127103526 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:25.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x56265014f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x56265014e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 -- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 msgr2=0x56265014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x56265014f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 -- 172.21.15.150:0/127103526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 msgr2=0x56265014e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x56265014e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.664+0000 7fa91affd640 1 -- 172.21.15.150:0/127103526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264fdc1210 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa91affd640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa90c02c630 tx=0x7fa90c0093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce12e9a93f0c69b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039050 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa90c032790 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa90c032c90 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa90c032990 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 10 client.15522 fetch_fsmap learned FSMap version 23 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56264fe8d620 con 0x562650148a50 2024-03-18T16:01:25.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c032c90 con 0x562650148a50 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa90c03f880 con 0x562650148a50 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 10 client.15522 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 10 client.15522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 20 client.15522 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501617e0 0x562650163bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 4 client.15522 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.665+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626500b9f80 con 0x5626501617e0 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.666+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501617e0 0x562650163bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.666+0000 7fa8faffd640 10 client.15522 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.666+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501617e0 0x562650163bd0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8764a0cf2162bf42 server_cookie=781e4dc9ba929dd9 in_seq=0 out_seq=0 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa8faffd640 1 -- 172.21.15.150:0/127103526 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626500b9f80 con 0x5626501617e0 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa8faffd640 10 client.15522 handle_command_reply: tid=0 2024-03-18T16:01:25.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 client.15522 shutdown 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 msgr2=0x56265014ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa90c02c630 tx=0x7fa90c0093e0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 shutdown_connections 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501617e0 0x562650163bd0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x56265014f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265014ed20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 --2- 172.21.15.150:0/127103526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148130 0x56265014e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa8faffd640 0 client.15522 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 >> 172.21.15.150:0/127103526 conn(0x562650142a50 msgr2=0x5626501447a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 shutdown_connections 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 1 -- 172.21.15.150:0/127103526 wait complete. 2024-03-18T16:01:25.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.667+0000 7fa92458d740 20 client.15522 trim_cache size 0 max 16384 2024-03-18T16:01:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 Processor -- start 2024-03-18T16:01:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 -- start start 2024-03-18T16:01:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650148130 0x562650150f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650151820 0x562650169170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1afb90 con 0x562650148a50 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ad460 con 0x562650151820 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa92458d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ab2b0 con 0x562650148130 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35822/0 (socket says 172.21.15.150:35822) 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 -- 172.21.15.150:0/2012896932 learned_addr learned my addr 172.21.15.150:0/2012896932 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650148130 0x562650150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650151820 0x562650169170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 -- 172.21.15.150:0/2012896932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650148130 msgr2=0x562650150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650148130 0x562650150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 -- 172.21.15.150:0/2012896932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650151820 msgr2=0x562650169170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650151820 0x562650169170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 -- 172.21.15.150:0/2012896932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56265015d700 con 0x562650148a50 2024-03-18T16:01:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.682+0000 7fa91affd640 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa90c03fdf0 tx=0x7fa90c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7e82af3776848 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa9017fa640 1 -- 172.21.15.150:0/2012896932 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c043070 con 0x562650148a50 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa9017fa640 1 -- 172.21.15.150:0/2012896932 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c005bd0 con 0x562650148a50 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa9017fa640 1 -- 172.21.15.150:0/2012896932 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039040 con 0x562650148a50 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 -- 172.21.15.150:0/2012896932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 msgr2=0x56265015d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa90c03fdf0 tx=0x7fa90c000f40 comp rx=0 tx=0).stop 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 -- 172.21.15.150:0/2012896932 shutdown_connections 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650151820 0x562650169170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148a50 0x56265015d1c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 --2- 172.21.15.150:0/2012896932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650148130 0x562650150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 -- 172.21.15.150:0/2012896932 >> 172.21.15.150:0/2012896932 conn(0x562650142a50 msgr2=0x56265014f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 -- 172.21.15.150:0/2012896932 shutdown_connections 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.683+0000 7fa92458d740 1 -- 172.21.15.150:0/2012896932 wait complete. 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 -- start start 2024-03-18T16:01:25.689 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 Processor -- start 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 0x562650155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 0x562650155660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ab920 con 0x562650148130 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1acd90 con 0x562650148a50 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1abff0 con 0x562650151820 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa92458d740 10 client.0 _ll_register_callbacks cb 0x7fa914009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 0x562650155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 0x562650155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 0x562650155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35830/0 (socket says 172.21.15.150:35830) 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91b7fe640 1 -- 172.21.15.150:0/3301960870 learned_addr learned my addr 172.21.15.150:0/3301960870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 -- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 msgr2=0x562650155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.690 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 -- 172.21.15.150:0/3301960870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 msgr2=0x562650155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 0x562650155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91affd640 1 -- 172.21.15.150:0/3301960870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626501c67d0 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.684+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa91affd640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 0x562650155660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa90c003d10 tx=0x7fa90c03f6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=125e22b7fafa85ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c039030 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa90c005bd0 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fa90c0040f0 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa90c043040 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 10 client.25301 fetch_fsmap learned FSMap version 23 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5626501c7ac0 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c005bd0 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa90c06f510 con 0x562650148a50 2024-03-18T16:01:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 10 client.25301 fetch_fsmap finished waiting for FSMap version 23 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 10 client.25301 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 20 client.25301 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501c80b0 0x5626501ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 4 client.25301 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.685+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626500b42d0 con 0x5626501c80b0 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.686+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501c80b0 0x5626501ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.686+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501c80b0 0x5626501ca4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e657d3ff89321f server_cookie=55e7f03784f93542 in_seq=0 out_seq=0 2024-03-18T16:01:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.686+0000 7fa8faffd640 10 client.25301 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa8faffd640 1 -- 172.21.15.150:0/3301960870 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626500b42d0 con 0x5626501c80b0 2024-03-18T16:01:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa8faffd640 10 client.25301 handle_command_reply: tid=0 2024-03-18T16:01:25.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa92458d740 1 client.25301 shutdown 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 msgr2=0x562650155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 0x562650155660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa90c003d10 tx=0x7fa90c03f6b0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.710+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 shutdown_connections 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5626501c80b0 0x5626501ca4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562650151820 0x562650155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562650148a50 0x562650155660 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa8faffd640 0 client.25301 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 --2- 172.21.15.150:0/3301960870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562650148130 0x562650155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 >> 172.21.15.150:0/3301960870 conn(0x562650142a50 msgr2=0x56265014e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 shutdown_connections 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 1 -- 172.21.15.150:0/3301960870 wait complete. 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa92458d740 20 client.25301 trim_cache size 0 max 16384 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:01:25.714 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:01:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 msgr2=0x7fa9040a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 0x7fa9040a5070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9100cc9a0 tx=0x7fa91006e460 comp rx=0 tx=0).stop 2024-03-18T16:01:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 msgr2=0x7fa91c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa91000ea60 tx=0x7fa91000ef20 comp rx=0 tx=0).stop 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 shutdown_connections 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa9040a2bb0 0x7fa9040a5070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa91c107860 0x7fa91c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c06a9a0 0x7fa91c1984e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 --2- 172.21.15.150:0/3968771746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa91c069fe0 0x7fa91c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 >> 172.21.15.150:0/3968771746 conn(0x7fa91c0774b0 msgr2=0x7fa91c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 shutdown_connections 2024-03-18T16:01:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:25.711+0000 7fa9219a9640 1 -- 172.21.15.150:0/3968771746 wait complete. 2024-03-18T16:01:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:25 smithi150 ceph-mon[31946]: pgmap v43: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:25 smithi191 ceph-mon[36784]: pgmap v43: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:25 smithi156 ceph-mon[35919]: pgmap v43: 97 pgs: 97 active+clean; 4.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 307 KiB/s wr, 41 op/s 2024-03-18T16:01:26.312 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10719 6.99 MB/sec warmup 11 sec latency 2092.616 ms 2024-03-18T16:01:27.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10735 6.41 MB/sec warmup 12 sec latency 2425.312 ms 2024-03-18T16:01:27.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:27 smithi150 ceph-mon[31946]: pgmap v44: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 525 KiB/s wr, 70 op/s 2024-03-18T16:01:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:27 smithi191 ceph-mon[36784]: pgmap v44: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 525 KiB/s wr, 70 op/s 2024-03-18T16:01:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:27 smithi156 ceph-mon[35919]: pgmap v44: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 525 KiB/s wr, 70 op/s 2024-03-18T16:01:28.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10805 5.92 MB/sec warmup 13 sec latency 935.305 ms 2024-03-18T16:01:28.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:28 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:28.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:28 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:29.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:28 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:29.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10805 5.50 MB/sec warmup 14 sec latency 1674.133 ms 2024-03-18T16:01:29.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:29 smithi150 ceph-mon[31946]: pgmap v45: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 514 KiB/s wr, 64 op/s 2024-03-18T16:01:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:29 smithi191 ceph-mon[36784]: pgmap v45: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 514 KiB/s wr, 64 op/s 2024-03-18T16:01:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:29 smithi156 ceph-mon[35919]: pgmap v45: 97 pgs: 97 active+clean; 6.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 514 KiB/s wr, 64 op/s 2024-03-18T16:01:30.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10805 5.13 MB/sec warmup 15 sec latency 2674.246 ms 2024-03-18T16:01:31.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 10805 4.81 MB/sec warmup 16 sec latency 3674.344 ms 2024-03-18T16:01:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:31 smithi191 ceph-mon[36784]: pgmap v46: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 531 KiB/s wr, 85 op/s 2024-03-18T16:01:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:31 smithi156 ceph-mon[35919]: pgmap v46: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 531 KiB/s wr, 85 op/s 2024-03-18T16:01:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:31 smithi150 ceph-mon[31946]: pgmap v46: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 531 KiB/s wr, 85 op/s 2024-03-18T16:01:32.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 11629 4.66 MB/sec warmup 17 sec latency 4012.627 ms 2024-03-18T16:01:33.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 14406 5.40 MB/sec warmup 18 sec latency 22.483 ms 2024-03-18T16:01:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:33 smithi191 ceph-mon[36784]: pgmap v47: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:33 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:33 smithi156 ceph-mon[35919]: pgmap v47: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:33 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:33 smithi150 ceph-mon[31946]: pgmap v47: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:33 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:01:34.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 16632 5.50 MB/sec warmup 19 sec latency 15.070 ms 2024-03-18T16:01:35.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 18481 5.84 MB/sec warmup 20 sec latency 12.829 ms 2024-03-18T16:01:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:35 smithi191 ceph-mon[36784]: pgmap v48: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:35 smithi156 ceph-mon[35919]: pgmap v48: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:35 smithi150 ceph-mon[31946]: pgmap v48: 97 pgs: 97 active+clean; 6.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 235 KiB/s wr, 49 op/s 2024-03-18T16:01:36.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 21453 6.40 MB/sec warmup 21 sec latency 16.818 ms 2024-03-18T16:01:36.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:36 smithi191 ceph-mon[36784]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-18T16:01:37.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:36 smithi156 ceph-mon[35919]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-18T16:01:37.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:36 smithi150 ceph-mon[31946]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-18T16:01:37.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 23679 6.48 MB/sec warmup 22 sec latency 14.347 ms 2024-03-18T16:01:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:37 smithi191 ceph-mon[36784]: pgmap v49: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 149 op/s 2024-03-18T16:01:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:37 smithi156 ceph-mon[35919]: pgmap v49: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 149 op/s 2024-03-18T16:01:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:37 smithi150 ceph-mon[31946]: pgmap v49: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 149 op/s 2024-03-18T16:01:38.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 25541 6.74 MB/sec warmup 23 sec latency 19.377 ms 2024-03-18T16:01:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 -- 172.21.15.150:0/2041800715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8105680 msgr2=0x7f9fe8105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 --2- 172.21.15.150:0/2041800715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8105680 0x7f9fe8105a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009900 tx=0x7f9fd802bc30 comp rx=0 tx=0).stop 2024-03-18T16:01:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 -- 172.21.15.150:0/2041800715 shutdown_connections 2024-03-18T16:01:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 --2- 172.21.15.150:0/2041800715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe81012f0 0x7f9fe810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 --2- 172.21.15.150:0/2041800715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe8100930 0x7f9fe8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 --2- 172.21.15.150:0/2041800715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8105680 0x7f9fe8105a60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 -- 172.21.15.150:0/2041800715 >> 172.21.15.150:0/2041800715 conn(0x7f9fe8077530 msgr2=0x7f9fe8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 -- 172.21.15.150:0/2041800715 shutdown_connections 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 -- 172.21.15.150:0/2041800715 wait complete. 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.807+0000 7f9fed148640 1 Processor -- start 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 -- start start 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe81012f0 0x7f9fe81974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe8105680 0x7f9fe819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8110bb0 con 0x7f9fe8100930 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8110a30 con 0x7f9fe8105680 2024-03-18T16:01:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fed148640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8110d30 con 0x7f9fe81012f0 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37644/0 (socket says 172.21.15.150:37644) 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/3873986784 learned_addr learned my addr 172.21.15.150:0/3873986784 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fdffff640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe81012f0 0x7f9fe81974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe759e640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe8105680 0x7f9fe819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/3873986784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe81012f0 msgr2=0x7f9fe81974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe81012f0 0x7f9fe81974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/3873986784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe8105680 msgr2=0x7f9fe819b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe8105680 0x7f9fe819b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/3873986784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe819bf80 con 0x7f9fe8100930 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9fd802efd0 tx=0x7f9fd8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.808+0000 7f9fe4d99640 1 -- 172.21.15.150:0/3873986784 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8041070 con 0x7f9fe8100930 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.809+0000 7f9fe4d99640 1 -- 172.21.15.150:0/3873986784 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd80054e0 con 0x7f9fe8100930 2024-03-18T16:01:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.809+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe819c210 con 0x7f9fe8100930 2024-03-18T16:01:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.809+0000 7f9fe4d99640 1 -- 172.21.15.150:0/3873986784 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8003d40 con 0x7f9fe8100930 2024-03-18T16:01:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.809+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81a3b00 con 0x7f9fe8100930 2024-03-18T16:01:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.811+0000 7f9fe4d99640 1 -- 172.21.15.150:0/3873986784 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f9fd8044070 con 0x7f9fe8100930 2024-03-18T16:01:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.811+0000 7f9fefe0b740 1 Processor -- start 2024-03-18T16:01:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.811+0000 7f9fefe0b740 1 -- start start 2024-03-18T16:01:38.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.811+0000 7f9fe4d99640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 0x7f9fd40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe759e640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 0x7f9fd40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe4d99640 1 -- 172.21.15.150:0/3873986784 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f9fd80f0770 con 0x7f9fe8100930 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d7a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d7ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559874cd3a40 con 0x559874d6ec50 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559874c7c920 con 0x559874d7aa30 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559874ccfae0 con 0x559874d6f570 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37658/0 (socket says 172.21.15.150:37658) 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/669377013 learned_addr learned my addr 172.21.15.150:0/669377013 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fdffff640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d7a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe759e640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d7ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/669377013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 msgr2=0x559874d7a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d7a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/669377013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 msgr2=0x559874d7ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d7ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 -- 172.21.15.150:0/669377013 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559874c94460 con 0x559874d6ec50 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe759e640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d7ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe759e640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 0x7f9fd40a4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005fd0 tx=0x7f9fe0011810 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80ee510 tx=0x7f9fd802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7c46857408c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fdd7fa640 1 -- 172.21.15.150:0/669377013 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd804b040 con 0x559874d6ec50 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fdd7fa640 1 -- 172.21.15.150:0/669377013 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8004c80 con 0x559874d6ec50 2024-03-18T16:01:38.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.812+0000 7f9fdd7fa640 1 -- 172.21.15.150:0/669377013 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80b9650 con 0x559874d6ec50 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 -- 172.21.15.150:0/669377013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 msgr2=0x559874d6f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80ee510 tx=0x7f9fd802c7e0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 -- 172.21.15.150:0/669377013 shutdown_connections 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d7ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d7a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/669377013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d6f030 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 -- 172.21.15.150:0/669377013 >> 172.21.15.150:0/669377013 conn(0x559874d69a50 msgr2=0x559874d69e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 -- 172.21.15.150:0/669377013 shutdown_connections 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.813+0000 7f9fefe0b740 1 -- 172.21.15.150:0/669377013 wait complete. 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 -- start start 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 Processor -- start 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6ec50 0x559874d77ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d78530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559874d7d3b0 con 0x559874d6f570 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559874c8fd70 con 0x559874d7aa30 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559874d7d530 con 0x559874d6ec50 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fefe0b740 10 client.0 _ll_register_callbacks cb 0x5598749e7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37672/0 (socket says 172.21.15.150:37672) 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 -- 172.21.15.150:0/2767197380 learned_addr learned my addr 172.21.15.150:0/2767197380 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6ec50 0x559874d77ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fe759e640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d78530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 -- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6ec50 msgr2=0x559874d77ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6ec50 0x559874d77ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 -- 172.21.15.150:0/2767197380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 msgr2=0x559874d78530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d78530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 -- 172.21.15.150:0/2767197380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598749e8210 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0011440 tx=0x7f9fd0011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a232cfc963069eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.814+0000 7f9fce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0004020 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f9fd00041c0 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd8004c80 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9fd00043c0 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 10 client.15540 fetch_fsmap learned FSMap version 25 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559874ab4620 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8004c80 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9fd0003ae0 con 0x559874d6f570 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 10 client.15540 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:01:38.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 10 client.15540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 20 client.15540 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874d8a9d0 0x559874d8cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 4 client.15540 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559874ce0f80 con 0x559874d8a9d0 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.815+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874d8a9d0 0x559874d8cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.816+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874d8a9d0 0x559874d8cde0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=222522c85f890c22 server_cookie=b104fc7eb8a87ad5 in_seq=0 out_seq=0 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.816+0000 7f9fce7fc640 10 client.15540 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2767197380 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559874ce0f80 con 0x559874d8a9d0 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fce7fc640 10 client.15540 handle_command_reply: tid=0 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 client.15540 shutdown 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 msgr2=0x559874d77ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0011440 tx=0x7f9fd0011900 comp rx=0 tx=0).stop 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 shutdown_connections 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874d8a9d0 0x559874d8cde0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d78530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fce7fc640 0 client.15540 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d77ff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2767197380 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6ec50 0x559874d77ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 >> 172.21.15.150:0/2767197380 conn(0x559874d69a50 msgr2=0x559874d6b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 shutdown_connections 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2767197380 wait complete. 2024-03-18T16:01:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.817+0000 7f9fefe0b740 20 client.15540 trim_cache size 0 max 16384 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.832+0000 7f9fefe0b740 1 Processor -- start 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.832+0000 7f9fefe0b740 1 -- start start 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d86150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81adaa0 con 0x559874d6ec50 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81ab310 con 0x559874d7aa30 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81a9180 con 0x559874d6f570 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:37184/0 (socket says 172.21.15.150:37184) 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 -- 172.21.15.150:0/1246599766 learned_addr learned my addr 172.21.15.150:0/1246599766 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:38.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37688/0 (socket says 172.21.15.150:37688) 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d86150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 -- 172.21.15.150:0/1246599766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 msgr2=0x559874d86150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d86150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 -- 172.21.15.150:0/1246599766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 msgr2=0x559874d82b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 -- 172.21.15.150:0/1246599766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559874d77880 con 0x559874d7aa30 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fe759e640 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0003ed0 tx=0x7f9fe001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911c755582470b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.833+0000 7f9fdcff9640 1 -- 172.21.15.150:0/1246599766 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0017040 con 0x559874d7aa30 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fdcff9640 1 -- 172.21.15.150:0/1246599766 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0019070 con 0x559874d7aa30 2024-03-18T16:01:38.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fdcff9640 1 -- 172.21.15.150:0/1246599766 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0005240 con 0x559874d7aa30 2024-03-18T16:01:38.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 -- 172.21.15.150:0/1246599766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 msgr2=0x559874d76380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0003ed0 tx=0x7f9fe001d000 comp rx=0 tx=0).stop 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 -- 172.21.15.150:0/1246599766 shutdown_connections 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d7aa30 0x559874d76380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d6f570 0x559874d86150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/1246599766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6ec50 0x559874d82b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 -- 172.21.15.150:0/1246599766 >> 172.21.15.150:0/1246599766 conn(0x559874d69a50 msgr2=0x559874d64af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 -- 172.21.15.150:0/1246599766 shutdown_connections 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.834+0000 7f9fefe0b740 1 -- 172.21.15.150:0/1246599766 wait complete. 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 -- start start 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 Processor -- start 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 0x559874d751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d7aa30 0x559874d73d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81a97d0 con 0x559874d6f570 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81aac40 con 0x559874d6ec50 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81a9ea0 con 0x559874d7aa30 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fefe0b740 10 client.0 _ll_register_callbacks cb 0x7f9fd000bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37690/0 (socket says 172.21.15.150:37690) 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 -- 172.21.15.150:0/2846430283 learned_addr learned my addr 172.21.15.150:0/2846430283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 0x559874d751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fe759e640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d7aa30 0x559874d73d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 -- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d7aa30 msgr2=0x559874d73d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d7aa30 0x559874d73d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 -- 172.21.15.150:0/2846430283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 msgr2=0x559874d751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:38.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 0x559874d751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 -- 172.21.15.150:0/2846430283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559874d74570 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 0x559874d751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fdffff640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0005d00 tx=0x7f9fd0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9338519ae00c4a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.835+0000 7f9fce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0030020 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f9fd00055f0 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fe0019070 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9fd003b050 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 10 client.15546 fetch_fsmap learned FSMap version 25 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559874df2890 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0019070 con 0x559874d6f570 2024-03-18T16:01:38.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9fd0010ad0 con 0x559874d6f570 2024-03-18T16:01:38.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 10 client.15546 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:01:38.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 10 client.15546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:38.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 20 client.15546 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:38.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874df3000 0x559874df5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 4 client.15546 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:01:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559874cdb2d0 con 0x559874df3000 2024-03-18T16:01:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.836+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874df3000 0x559874df5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.837+0000 7f9fe6d9d640 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874df3000 0x559874df5420 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6123db8b6deee8d server_cookie=9decfa286a8576be in_seq=0 out_seq=0 2024-03-18T16:01:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.837+0000 7f9fce7fc640 10 client.15546 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.847+0000 7f9fce7fc640 1 -- 172.21.15.150:0/2846430283 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559874cdb2d0 con 0x559874df3000 2024-03-18T16:01:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.847+0000 7f9fce7fc640 10 client.15546 handle_command_reply: tid=0 2024-03-18T16:01:38.849 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.847+0000 7f9fefe0b740 1 client.15546 shutdown 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 msgr2=0x559874d756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0005d00 tx=0x7f9fd0005270 comp rx=0 tx=0).stop 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 shutdown_connections 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559874df3000 0x559874df5420 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559874d7aa30 0x559874d73d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559874d6f570 0x559874d756e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fce7fc640 0 client.15546 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 --2- 172.21.15.150:0/2846430283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559874d6ec50 0x559874d751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 >> 172.21.15.150:0/2846430283 conn(0x559874d69a50 msgr2=0x559874d76340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 shutdown_connections 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 1 -- 172.21.15.150:0/2846430283 wait complete. 2024-03-18T16:01:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fefe0b740 20 client.15546 trim_cache size 0 max 16384 2024-03-18T16:01:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:01:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:01:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:01:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 msgr2=0x7f9fd40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 0x7f9fd40a4f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005fd0 tx=0x7f9fe0011810 comp rx=0 tx=0).stop 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 msgr2=0x7f9fe8196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9fd802efd0 tx=0x7f9fd8004430 comp rx=0 tx=0).stop 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 shutdown_connections 2024-03-18T16:01:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9fd40a2a90 0x7f9fd40a4f50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fe8105680 0x7f9fe819b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fe81012f0 0x7f9fe81974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 --2- 172.21.15.150:0/3873986784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe8100930 0x7f9fe8196f90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 >> 172.21.15.150:0/3873986784 conn(0x7f9fe8077530 msgr2=0x7f9fe80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 shutdown_connections 2024-03-18T16:01:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:38.848+0000 7f9fed148640 1 -- 172.21.15.150:0/3873986784 wait complete. 2024-03-18T16:01:39.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 28479 7.19 MB/sec warmup 24 sec latency 25.737 ms 2024-03-18T16:01:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:39 smithi191 ceph-mon[36784]: pgmap v50: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-03-18T16:01:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:39 smithi156 ceph-mon[35919]: pgmap v50: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-03-18T16:01:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:39 smithi150 ceph-mon[31946]: pgmap v50: 97 pgs: 97 active+clean; 29 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-03-18T16:01:40.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 30397 7.18 MB/sec warmup 25 sec latency 16.989 ms 2024-03-18T16:01:41.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 32544 7.43 MB/sec warmup 26 sec latency 16.190 ms 2024-03-18T16:01:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:41 smithi191 ceph-mon[36784]: pgmap v51: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-18T16:01:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:41 smithi156 ceph-mon[35919]: pgmap v51: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-18T16:01:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:41 smithi150 ceph-mon[31946]: pgmap v51: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-18T16:01:42.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 35326 7.69 MB/sec warmup 27 sec latency 41.285 ms 2024-03-18T16:01:43.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 37203 7.77 MB/sec warmup 28 sec latency 11.852 ms 2024-03-18T16:01:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:43 smithi191 ceph-mon[36784]: pgmap v52: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:43 smithi156 ceph-mon[35919]: pgmap v52: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:43 smithi150 ceph-mon[31946]: pgmap v52: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:44.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 39585 7.98 MB/sec warmup 29 sec latency 20.495 ms 2024-03-18T16:01:45.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 42405 8.19 MB/sec warmup 30 sec latency 15.546 ms 2024-03-18T16:01:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:45 smithi191 ceph-mon[36784]: pgmap v53: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:45 smithi156 ceph-mon[35919]: pgmap v53: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:45 smithi150 ceph-mon[31946]: pgmap v53: 97 pgs: 97 active+clean; 35 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-18T16:01:46.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 44256 8.25 MB/sec warmup 31 sec latency 18.280 ms 2024-03-18T16:01:47.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 46616 8.42 MB/sec warmup 32 sec latency 13.213 ms 2024-03-18T16:01:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:47 smithi191 ceph-mon[36784]: pgmap v54: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-18T16:01:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:47 smithi156 ceph-mon[35919]: pgmap v54: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-18T16:01:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:47 smithi150 ceph-mon[31946]: pgmap v54: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-18T16:01:48.313 INFO:tasks.workunit.client.0.smithi150.stdout: 1 49332 8.57 MB/sec warmup 33 sec latency 17.191 ms 2024-03-18T16:01:49.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 51385 8.65 MB/sec warmup 34 sec latency 10.927 ms 2024-03-18T16:01:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:49 smithi191 ceph-mon[36784]: pgmap v55: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 178 op/s 2024-03-18T16:01:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:49 smithi156 ceph-mon[35919]: pgmap v55: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 178 op/s 2024-03-18T16:01:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:49 smithi150 ceph-mon[31946]: pgmap v55: 97 pgs: 97 active+clean; 41 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.2 MiB/s wr, 178 op/s 2024-03-18T16:01:50.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 53712 8.79 MB/sec warmup 35 sec latency 15.179 ms 2024-03-18T16:01:51.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 56444 8.92 MB/sec warmup 36 sec latency 24.002 ms 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/922460987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4105680 msgr2=0x7ff7c4105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/922460987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4105680 0x7ff7c4105a60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009900 tx=0x7ff7c002bc30 comp rx=0 tx=0).stop 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/922460987 shutdown_connections 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/922460987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 0x7ff7c410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/922460987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4100930 0x7ff7c4100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/922460987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4105680 0x7ff7c4105a60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/922460987 >> 172.21.15.150:0/922460987 conn(0x7ff7c4077530 msgr2=0x7ff7c4077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/922460987 shutdown_connections 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.936+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/922460987 wait complete. 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 Processor -- start 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 -- start start 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 0x7ff7c41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105680 0x7ff7c419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4110ba0 con 0x7ff7c4100930 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4110a20 con 0x7ff7c4105680 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7cc4f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c4110d20 con 0x7ff7c41012f0 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58474/0 (socket says 172.21.15.150:58474) 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 -- 172.21.15.150:0/2110600229 learned_addr learned my addr 172.21.15.150:0/2110600229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 0x7ff7c41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105680 0x7ff7c419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 -- 172.21.15.150:0/2110600229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 msgr2=0x7ff7c41985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 0x7ff7c41985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 -- 172.21.15.150:0/2110600229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105680 msgr2=0x7ff7c419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105680 0x7ff7c419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 -- 172.21.15.150:0/2110600229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c419d090 con 0x7ff7c4100930 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.937+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005940 tx=0x7ff7c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.938+0000 7ff7bb7fe640 1 -- 172.21.15.150:0/2110600229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0041070 con 0x7ff7c4100930 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.938+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c419d320 con 0x7ff7c4100930 2024-03-18T16:01:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.938+0000 7ff7bb7fe640 1 -- 172.21.15.150:0/2110600229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c00054e0 con 0x7ff7c4100930 2024-03-18T16:01:51.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.938+0000 7ff7bb7fe640 1 -- 172.21.15.150:0/2110600229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0003d00 con 0x7ff7c4100930 2024-03-18T16:01:51.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.938+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c41a4c10 con 0x7ff7c4100930 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.940+0000 7ff7bb7fe640 1 -- 172.21.15.150:0/2110600229 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7ff7c0044070 con 0x7ff7c4100930 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7bb7fe640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 0x7ff7b00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 Processor -- start 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7bb7fe640 1 -- 172.21.15.150:0/2110600229 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff7c00f0940 con 0x7ff7c4100930 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 0x7ff7b00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 -- start start 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f11570 0x55a551f1e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a551e75a40 con 0x55a551f1ec30 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a551e1e920 con 0x55a551f11570 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a551e71ae0 con 0x55a551f10c50 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 0x7ff7b00a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4077b50 tx=0x7ff7b4005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7caa8e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7caa8e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58488/0 (socket says 172.21.15.150:58488) 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:34558/0 (socket says 172.21.15.150:34558) 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7caa8e640 1 -- 172.21.15.150:0/839280768 learned_addr learned my addr 172.21.15.150:0/839280768 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.941+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f11570 0x55a551f1e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 -- 172.21.15.150:0/839280768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 msgr2=0x55a551f11030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 -- 172.21.15.150:0/839280768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f11570 msgr2=0x55a551f1e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f11570 0x55a551f1e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 -- 172.21.15.150:0/839280768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a551e36460 con 0x55a551f1ec30 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009fc0 tx=0x7ff7bc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b56cdbff28d018 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7b8bfe640 1 -- 172.21.15.150:0/839280768 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc004d20 con 0x55a551f1ec30 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7b8bfe640 1 -- 172.21.15.150:0/839280768 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7bc004050 con 0x55a551f1ec30 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.942+0000 7ff7b8bfe640 1 -- 172.21.15.150:0/839280768 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc005280 con 0x55a551f1ec30 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/839280768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 msgr2=0x55a551f21020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009fc0 tx=0x7ff7bc0118d0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/839280768 shutdown_connections 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f1ec30 0x55a551f21020 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f11570 0x55a551f1e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/839280768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f11030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/839280768 >> 172.21.15.150:0/839280768 conn(0x55a551f0ba50 msgr2=0x55a551f0be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/839280768 shutdown_connections 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/839280768 wait complete. 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- start start 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 Processor -- start 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f1bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 0x55a551f1c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a551f217f0 con 0x55a551f11570 2024-03-18T16:01:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a551e31d70 con 0x55a551f1ec30 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a551f21970 con 0x55a551f10c50 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7cf1bc740 10 client.0 _ll_register_callbacks cb 0x55a551b89a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.943+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f1bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58498/0 (socket says 172.21.15.150:58498) 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/1300353520 learned_addr learned my addr 172.21.15.150:0/1300353520 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 0x55a551f1c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 msgr2=0x55a551f1bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f1bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/1300353520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 msgr2=0x55a551f1c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 0x55a551f1c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/1300353520 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a551b8a210 con 0x55a551f11570 2024-03-18T16:01:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f1bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 0x55a551f1c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40038a0 tx=0x7ff7b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8b359970ac6209 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7adffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b400e6a0 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff7b400e840 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7bc004050 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff7b400ea40 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7cf1bc740 10 client.15558 fetch_fsmap learned FSMap version 25 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a551c56620 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.944+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7bc004050 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7ff7b4027c90 con 0x55a551f11570 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 10 client.15558 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:01:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 20 client.15558 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f2eec0 0x55a551f312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 4 client.15558 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a551e82f80 con 0x55a551f2eec0 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f2eec0 0x55a551f312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7adffb640 10 client.15558 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.945+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f2eec0 0x55a551f312e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9418dd3308636d32 server_cookie=62bdf8fac153d570 in_seq=0 out_seq=0 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7adffb640 1 -- 172.21.15.150:0/1300353520 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a551e82f80 con 0x55a551f2eec0 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7adffb640 10 client.15558 handle_command_reply: tid=0 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 client.15558 shutdown 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 msgr2=0x55a551f1c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40038a0 tx=0x7ff7b4004730 comp rx=0 tx=0).stop 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 shutdown_connections 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f2eec0 0x55a551f312e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f1ec30 0x55a551f1c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f1c0d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7adffb640 0 client.15558 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/1300353520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f10c50 0x55a551f1bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.946+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 >> 172.21.15.150:0/1300353520 conn(0x55a551f0ba50 msgr2=0x55a551f0d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.947+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 shutdown_connections 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.947+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/1300353520 wait complete. 2024-03-18T16:01:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.947+0000 7ff7cf1bc740 20 client.15558 trim_cache size 0 max 16384 2024-03-18T16:01:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:51 smithi191 ceph-mon[36784]: pgmap v56: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 264 op/s 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 Processor -- start 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 -- start start 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f1e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f36bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41ada40 con 0x55a551f11570 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41ab2b0 con 0x55a551f10c50 2024-03-18T16:01:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41a9120 con 0x55a551f1ec30 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58508/0 (socket says 172.21.15.150:58508) 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/3400789148 learned_addr learned my addr 172.21.15.150:0/3400789148 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f1e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7caa8e640 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f36bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/3400789148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 msgr2=0x55a551f36bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.963+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f36bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/3400789148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 msgr2=0x55a551f1e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f1e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/3400789148 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a551f0b880 con 0x55a551f11570 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff7b401d6d0 tx=0x7ff7b4009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3ff4f6eeda03b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7affff640 1 -- 172.21.15.150:0/3400789148 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4034020 con 0x55a551f11570 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7affff640 1 -- 172.21.15.150:0/3400789148 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b40044d0 con 0x55a551f11570 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.964+0000 7ff7affff640 1 -- 172.21.15.150:0/3400789148 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b400ed10 con 0x55a551f11570 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/3400789148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 msgr2=0x55a551f2a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff7b401d6d0 tx=0x7ff7b4009640 comp rx=0 tx=0).stop 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/3400789148 shutdown_connections 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f36bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f2a630 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/3400789148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f1e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/3400789148 >> 172.21.15.150:0/3400789148 conn(0x55a551f0ba50 msgr2=0x55a551f0d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/3400789148 shutdown_connections 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/3400789148 wait complete. 2024-03-18T16:01:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- start start 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 Processor -- start 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f22b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f94220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41a9770 con 0x55a551f11570 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41aabe0 con 0x55a551f10c50 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c41a9e40 con 0x55a551f1ec30 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7cf1bc740 10 client.0 _ll_register_callbacks cb 0x7ff7b4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7ca28d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f22b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7c9a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58520/0 (socket says 172.21.15.150:58520) 2024-03-18T16:01:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/439463770 learned_addr learned my addr 172.21.15.150:0/439463770 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.965+0000 7ff7caa8e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f94220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 msgr2=0x55a551f94220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f94220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/439463770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 msgr2=0x55a551f22b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f22b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 -- 172.21.15.150:0/439463770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a551f94760 con 0x55a551f11570 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f22b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7c9a8c640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40058c0 tx=0x7ff7b400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac0918a06267ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7adffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b403d070 con 0x55a551f11570 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7ff7b4005390 con 0x55a551f11570 2024-03-18T16:01:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7b40044d0 con 0x55a551f11570 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff7b4005590 con 0x55a551f11570 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7cf1bc740 10 client.15564 fetch_fsmap learned FSMap version 25 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a551f95970 con 0x55a551f11570 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.966+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b40044d0 con 0x55a551f11570 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7ff7b4004ca0 con 0x55a551f11570 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 10 client.15564 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 10 client.15564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 20 client.15564 populate_metadata read hostname 'smithi150' 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f95fe0 0x55a551f98400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 4 client.15564 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a551e7d2d0 con 0x55a551f95fe0 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f95fe0 0x55a551f98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7ca28d640 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f95fe0 0x55a551f98400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26039585528cf1bc server_cookie=827d92057a07577c in_seq=0 out_seq=0 2024-03-18T16:01:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.967+0000 7ff7adffb640 10 client.15564 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7adffb640 1 -- 172.21.15.150:0/439463770 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a551e7d2d0 con 0x55a551f95fe0 2024-03-18T16:01:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7adffb640 10 client.15564 handle_command_reply: tid=0 2024-03-18T16:01:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 client.15564 shutdown 2024-03-18T16:01:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 msgr2=0x55a551f230d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40058c0 tx=0x7ff7b400eef0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 shutdown_connections 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a551f95fe0 0x55a551f98400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a551f1ec30 0x55a551f94220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a551f11570 0x55a551f230d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7adffb640 0 client.15564 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 --2- 172.21.15.150:0/439463770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a551f10c50 0x55a551f22b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.975+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 >> 172.21.15.150:0/439463770 conn(0x55a551f0ba50 msgr2=0x55a551f0a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 shutdown_connections 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cf1bc740 1 -- 172.21.15.150:0/439463770 wait complete. 2024-03-18T16:01:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cf1bc740 20 client.15564 trim_cache size 0 max 16384 2024-03-18T16:01:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:01:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:01:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:01:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 msgr2=0x7ff7b00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 0x7ff7b00a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4077b50 tx=0x7ff7b4005f90 comp rx=0 tx=0).stop 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 msgr2=0x7ff7c41980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005940 tx=0x7ff7c0004270 comp rx=0 tx=0).stop 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 shutdown_connections 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff7b00a2a10 0x7ff7b00a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7c4105680 0x7ff7c419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7c41012f0 0x7ff7c41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 --2- 172.21.15.150:0/2110600229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4100930 0x7ff7c41980a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 >> 172.21.15.150:0/2110600229 conn(0x7ff7c4077530 msgr2=0x7ff7c40690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 shutdown_connections 2024-03-18T16:01:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:01:51.976+0000 7ff7cc4f1640 1 -- 172.21.15.150:0/2110600229 wait complete. 2024-03-18T16:01:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:51 smithi156 ceph-mon[35919]: pgmap v56: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 264 op/s 2024-03-18T16:01:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:51 smithi150 ceph-mon[31946]: pgmap v56: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 264 op/s 2024-03-18T16:01:52.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 58338 8.97 MB/sec warmup 37 sec latency 13.017 ms 2024-03-18T16:01:53.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 60739 9.11 MB/sec warmup 38 sec latency 14.727 ms 2024-03-18T16:01:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:53 smithi191 ceph-mon[36784]: pgmap v57: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:53 smithi156 ceph-mon[35919]: pgmap v57: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:53 smithi150 ceph-mon[31946]: pgmap v57: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:54.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 63206 9.16 MB/sec warmup 39 sec latency 70.327 ms 2024-03-18T16:01:55.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 64928 9.18 MB/sec warmup 40 sec latency 66.402 ms 2024-03-18T16:01:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:55 smithi191 ceph-mon[36784]: pgmap v58: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:55 smithi156 ceph-mon[35919]: pgmap v58: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:55 smithi150 ceph-mon[31946]: pgmap v58: 97 pgs: 97 active+clean; 46 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 172 op/s 2024-03-18T16:01:56.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 67677 9.37 MB/sec warmup 41 sec latency 10.994 ms 2024-03-18T16:01:57.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 69920 9.33 MB/sec warmup 42 sec latency 14.330 ms 2024-03-18T16:01:57.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:57 smithi191 ceph-mon[36784]: pgmap v59: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-18T16:01:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:57 smithi156 ceph-mon[35919]: pgmap v59: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-18T16:01:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:57 smithi150 ceph-mon[31946]: pgmap v59: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-18T16:01:58.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 71765 9.39 MB/sec warmup 43 sec latency 17.414 ms 2024-03-18T16:01:59.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 74673 9.57 MB/sec warmup 44 sec latency 15.873 ms 2024-03-18T16:01:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:01:59 smithi191 ceph-mon[36784]: pgmap v60: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-03-18T16:02:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:01:59 smithi156 ceph-mon[35919]: pgmap v60: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-03-18T16:02:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:01:59 smithi150 ceph-mon[31946]: pgmap v60: 97 pgs: 97 active+clean; 52 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-03-18T16:02:00.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 76747 9.52 MB/sec warmup 45 sec latency 13.744 ms 2024-03-18T16:02:01.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 78327 9.60 MB/sec warmup 46 sec latency 30.716 ms 2024-03-18T16:02:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:01 smithi191 ceph-mon[36784]: pgmap v61: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-18T16:02:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:01 smithi156 ceph-mon[35919]: pgmap v61: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-18T16:02:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:01 smithi150 ceph-mon[31946]: pgmap v61: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-18T16:02:02.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 80494 9.55 MB/sec warmup 47 sec latency 12.456 ms 2024-03-18T16:02:03.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 82302 9.60 MB/sec warmup 48 sec latency 16.712 ms 2024-03-18T16:02:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:03 smithi191 ceph-mon[36784]: pgmap v62: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:03 smithi156 ceph-mon[35919]: pgmap v62: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:03 smithi150 ceph-mon[31946]: pgmap v62: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:04.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 85143 9.76 MB/sec warmup 49 sec latency 39.006 ms 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 -- 172.21.15.150:0/82228036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4350102150 msgr2=0x7f435010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 --2- 172.21.15.150:0/82228036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4350102150 0x7f435010e660 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4348009cd0 tx=0x7f434802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 -- 172.21.15.150:0/82228036 shutdown_connections 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 --2- 172.21.15.150:0/82228036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4350102150 0x7f435010e660 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 --2- 172.21.15.150:0/82228036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 0x7f4350101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 --2- 172.21.15.150:0/82228036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43501077a0 0x7f4350107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 -- 172.21.15.150:0/82228036 >> 172.21.15.150:0/82228036 conn(0x7f43500fd2a0 msgr2=0x7f43500ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 -- 172.21.15.150:0/82228036 shutdown_connections 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.067+0000 7f435708a640 1 -- 172.21.15.150:0/82228036 wait complete. 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 Processor -- start 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 -- start start 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 0x7f4350197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 0x7f43501984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4350111bb0 con 0x7f43501077a0 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4350111a30 con 0x7f4350101790 2024-03-18T16:02:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f435708a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4350111d30 con 0x7f4350102150 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57324/0 (socket says 172.21.15.150:57324) 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f434ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 0x7f43501984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 -- 172.21.15.150:0/2243650463 learned_addr learned my addr 172.21.15.150:0/2243650463 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4354e26640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 0x7f4350197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 -- 172.21.15.150:0/2243650463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 msgr2=0x7f43501984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 0x7f43501984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 -- 172.21.15.150:0/2243650463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 msgr2=0x7f4350197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 0x7f4350197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 -- 172.21.15.150:0/2243650463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435019cf50 con 0x7f43501077a0 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f434ffff640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 0x7f43501984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.068+0000 7f4355627640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f43480348f0 tx=0x7f4348032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.069+0000 7f434dffb640 1 -- 172.21.15.150:0/2243650463 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348041020 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.069+0000 7f434dffb640 1 -- 172.21.15.150:0/2243650463 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4348032e80 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.069+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435019d1e0 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.069+0000 7f434dffb640 1 -- 172.21.15.150:0/2243650463 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348039040 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.069+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43501a4ad0 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.071+0000 7f434dffb640 1 -- 172.21.15.150:0/2243650463 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f4348044020 con 0x7f43501077a0 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.071+0000 7f4359c70740 1 Processor -- start 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.071+0000 7f4359c70740 1 -- start start 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f434dffb640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 0x7f43400a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e87e71fa40 con 0x55e87e7c4820 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e87e6c8920 con 0x55e87e7bba50 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4359c70740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e87e71bae0 con 0x55e87e7bb130 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f434ffff640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 0x7f43400a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f434dffb640 1 -- 172.21.15.150:0/2243650463 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f43480f0830 con 0x7f43501077a0 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f434ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:48928/0 (socket says 172.21.15.150:48928) 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 -- 172.21.15.150:0/669416933 learned_addr learned my addr 172.21.15.150:0/669416933 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4355627640 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 -- 172.21.15.150:0/669416933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 msgr2=0x55e87e7c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 -- 172.21.15.150:0/669416933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 msgr2=0x55e87e7c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 -- 172.21.15.150:0/669416933 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e87e6e0460 con 0x55e87e7bb130 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f434ffff640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 0x7f43400a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f433c009fd0 tx=0x7f433c009f60 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.072+0000 7f4354e26640 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4344009ca0 tx=0x7f4344011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c63fa66c6092bb0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f433affd640 1 -- 172.21.15.150:0/669416933 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344019620 con 0x55e87e7bb130 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f433affd640 1 -- 172.21.15.150:0/669416933 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4344019ca0 con 0x55e87e7bb130 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f433affd640 1 -- 172.21.15.150:0/669416933 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344018bf0 con 0x55e87e7bb130 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 -- 172.21.15.150:0/669416933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 msgr2=0x55e87e7bb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4344009ca0 tx=0x7f4344011810 comp rx=0 tx=0).stop 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 -- 172.21.15.150:0/669416933 shutdown_connections 2024-03-18T16:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 --2- 172.21.15.150:0/669416933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7bb510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 -- 172.21.15.150:0/669416933 >> 172.21.15.150:0/669416933 conn(0x55e87e7b5a50 msgr2=0x55e87e7b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 -- 172.21.15.150:0/669416933 shutdown_connections 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.073+0000 7f4359c70740 1 -- 172.21.15.150:0/669416933 wait complete. 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 -- start start 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 Processor -- start 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bba50 0x55e87e7c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e87e7c7450 con 0x55e87e7c4820 2024-03-18T16:02:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e87e6dbd70 con 0x55e87e7bb130 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e87e7c75d0 con 0x55e87e7bba50 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4359c70740 10 client.0 _ll_register_callbacks cb 0x55e87e433a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4354e26640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4354e26640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:57778/0 (socket says 172.21.15.150:57778) 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4354e26640 1 -- 172.21.15.150:0/3304053792 learned_addr learned my addr 172.21.15.150:0/3304053792 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f4355627640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.074+0000 7f434ffff640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bba50 0x55e87e7c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 -- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bba50 msgr2=0x55e87e7c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bba50 0x55e87e7c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 -- 172.21.15.150:0/3304053792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 msgr2=0x55e87e7c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 -- 172.21.15.150:0/3304053792 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e87e434210 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4354e26640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43440058c0 tx=0x7f4344018fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab9e4bd99cf7b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f432ffff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344004220 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4344019ca0 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f43440043c0 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f434402e020 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4359c70740 10 client.25349 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e87e500620 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.075+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4344019ca0 con 0x55e87e7bb130 2024-03-18T16:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f43440197f0 con 0x55e87e7bb130 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 10 client.25349 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 10 client.25349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 20 client.25349 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e7d4830 0x55e87e7d6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 4 client.25349 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e87e72cf80 con 0x55e87e7d4830 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f434ffff640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e7d4830 0x55e87e7d6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f432ffff640 10 client.25349 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.076+0000 7f434ffff640 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e7d4830 0x55e87e7d6c20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64713050f47319b0 server_cookie=6466fc4a38e07a9f in_seq=0 out_seq=0 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f432ffff640 1 -- 172.21.15.150:0/3304053792 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e87e72cf80 con 0x55e87e7d4830 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f432ffff640 10 client.25349 handle_command_reply: tid=0 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 client.25349 shutdown 2024-03-18T16:02:05.088 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 msgr2=0x55e87e7c17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43440058c0 tx=0x7f4344018fb0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 shutdown_connections 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e7d4830 0x55e87e7d6c20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bba50 0x55e87e7c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 --2- 172.21.15.150:0/3304053792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bb130 0x55e87e7c17a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f432ffff640 0 client.25349 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 >> 172.21.15.150:0/3304053792 conn(0x55e87e7b5a50 msgr2=0x55e87e7b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 shutdown_connections 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 1 -- 172.21.15.150:0/3304053792 wait complete. 2024-03-18T16:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.077+0000 7f4359c70740 20 client.25349 trim_cache size 0 max 16384 2024-03-18T16:02:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 Processor -- start 2024-03-18T16:02:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 -- start start 2024-03-18T16:02:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7c4820 0x55e87e7dc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43501ada10 con 0x55e87e7bba50 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43501ab280 con 0x55e87e7c4820 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.092+0000 7f4359c70740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43501a90f0 con 0x55e87e7bb130 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f4354e26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57344/0 (socket says 172.21.15.150:57344) 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f4354e26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:48944/0 (socket says 172.21.15.150:48944) 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 -- 172.21.15.150:0/2166894166 learned_addr learned my addr 172.21.15.150:0/2166894166 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f4355627640 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7c4820 0x55e87e7dc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 -- 172.21.15.150:0/2166894166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 msgr2=0x55e87e7c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 -- 172.21.15.150:0/2166894166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7c4820 msgr2=0x55e87e7dc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7c4820 0x55e87e7dc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 -- 172.21.15.150:0/2166894166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e87e7b5880 con 0x55e87e7bba50 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f434ffff640 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f433c0095f0 tx=0x7f433c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7587ffc711348553 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f433a7fc640 1 -- 172.21.15.150:0/2166894166 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c003ac0 con 0x55e87e7bba50 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f433a7fc640 1 -- 172.21.15.150:0/2166894166 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f433c003c60 con 0x55e87e7bba50 2024-03-18T16:02:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.093+0000 7f433a7fc640 1 -- 172.21.15.150:0/2166894166 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c030560 con 0x55e87e7bba50 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- 172.21.15.150:0/2166894166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 msgr2=0x55e87e7d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f433c0095f0 tx=0x7f433c01a040 comp rx=0 tx=0).stop 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- 172.21.15.150:0/2166894166 shutdown_connections 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7c4820 0x55e87e7dc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7bba50 0x55e87e7d01f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 --2- 172.21.15.150:0/2166894166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- 172.21.15.150:0/2166894166 >> 172.21.15.150:0/2166894166 conn(0x55e87e7b5a50 msgr2=0x55e87e7b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- 172.21.15.150:0/2166894166 shutdown_connections 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- 172.21.15.150:0/2166894166 wait complete. 2024-03-18T16:02:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 -- start start 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.094+0000 7f4359c70740 1 Processor -- start 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7be190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43501a9740 con 0x55e87e7c4820 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43501aabb0 con 0x55e87e7bba50 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43501a9e10 con 0x55e87e7bb130 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 10 client.0 _ll_register_callbacks cb 0x7f4348005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4354e26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57356/0 (socket says 172.21.15.150:57356) 2024-03-18T16:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f434ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7be190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 -- 172.21.15.150:0/2424268493 learned_addr learned my addr 172.21.15.150:0/2424268493 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 -- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 msgr2=0x55e87e7ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 -- 172.21.15.150:0/2424268493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 msgr2=0x55e87e7be190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7be190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 -- 172.21.15.150:0/2424268493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e87e839ad0 con 0x55e87e7c4820 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4355627640 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43480ee190 tx=0x7f43480dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b0413a6a551a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f432ffff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348034920 con 0x55e87e7c4820 2024-03-18T16:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f43480c5300 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f433c003c60 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43480bd6f0 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 10 client.15582 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.095+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e87e83ad00 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f433c003c60 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f434803c030 con 0x55e87e7c4820 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 10 client.15582 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 10 client.15582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 20 client.15582 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e83b4d0 0x55e87e83d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 4 client.15582 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:02:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e87e7272d0 con 0x55e87e83b4d0 2024-03-18T16:02:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4354e26640 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e83b4d0 0x55e87e83d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f4354e26640 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e83b4d0 0x55e87e83d8f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc3d0d71e78aab5a server_cookie=635c12e0d648048e in_seq=0 out_seq=0 2024-03-18T16:02:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.096+0000 7f432ffff640 10 client.15582 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f432ffff640 1 -- 172.21.15.150:0/2424268493 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e87e7272d0 con 0x55e87e83b4d0 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f432ffff640 10 client.15582 handle_command_reply: tid=0 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 client.15582 shutdown 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 msgr2=0x55e87e7be8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43480ee190 tx=0x7f43480dd040 comp rx=0 tx=0).stop 2024-03-18T16:02:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 shutdown_connections 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e87e83b4d0 0x55e87e83d8f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e87e7c4820 0x55e87e7be8d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e87e7bba50 0x55e87e7be190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 --2- 172.21.15.150:0/2424268493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e87e7bb130 0x55e87e7ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f432ffff640 0 client.15582 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 >> 172.21.15.150:0/2424268493 conn(0x55e87e7b5a50 msgr2=0x55e87e7c16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 shutdown_connections 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 1 -- 172.21.15.150:0/2424268493 wait complete. 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.104+0000 7f4359c70740 20 client.15582 trim_cache size 0 max 16384 2024-03-18T16:02:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:02:05.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:02:05.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:02:05.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:02:05.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 msgr2=0x7f43400a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 0x7f43400a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f433c009fd0 tx=0x7f433c009f60 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 msgr2=0x7f435019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f43480348f0 tx=0x7f4348032c90 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 shutdown_connections 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f43400a2a90 0x7f43400a4f50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43501077a0 0x7f435019c820 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4350102150 0x7f43501984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 --2- 172.21.15.150:0/2243650463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4350101790 0x7f4350197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 >> 172.21.15.150:0/2243650463 conn(0x7f43500fd2a0 msgr2=0x7f43500fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 shutdown_connections 2024-03-18T16:02:05.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:05.105+0000 7f435708a640 1 -- 172.21.15.150:0/2243650463 wait complete. 2024-03-18T16:02:05.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 86871 9.70 MB/sec warmup 50 sec latency 20.542 ms 2024-03-18T16:02:05.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:05 smithi150 ceph-mon[31946]: pgmap v63: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:05.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:05 smithi191 ceph-mon[36784]: pgmap v63: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:05 smithi156 ceph-mon[35919]: pgmap v63: 97 pgs: 97 active+clean; 57 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 206 op/s 2024-03-18T16:02:06.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 89194 9.79 MB/sec warmup 51 sec latency 13.059 ms 2024-03-18T16:02:07.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 91894 9.85 MB/sec warmup 52 sec latency 22.122 ms 2024-03-18T16:02:07.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:07 smithi150 ceph-mon[31946]: pgmap v64: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 MiB/s wr, 302 op/s 2024-03-18T16:02:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:07 smithi191 ceph-mon[36784]: pgmap v64: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 MiB/s wr, 302 op/s 2024-03-18T16:02:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:07 smithi156 ceph-mon[35919]: pgmap v64: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 MiB/s wr, 302 op/s 2024-03-18T16:02:08.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 93307 9.81 MB/sec warmup 53 sec latency 103.162 ms 2024-03-18T16:02:09.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 96026 9.94 MB/sec warmup 54 sec latency 28.048 ms 2024-03-18T16:02:09.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:09 smithi150 ceph-mon[31946]: pgmap v65: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-03-18T16:02:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:09 smithi191 ceph-mon[36784]: pgmap v65: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-03-18T16:02:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:09 smithi156 ceph-mon[35919]: pgmap v65: 97 pgs: 97 active+clean; 64 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-03-18T16:02:10.314 INFO:tasks.workunit.client.0.smithi150.stdout: 1 98271 9.90 MB/sec warmup 55 sec latency 12.001 ms 2024-03-18T16:02:11.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 100079 9.94 MB/sec warmup 56 sec latency 15.720 ms 2024-03-18T16:02:11.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:11 smithi150 ceph-mon[31946]: pgmap v66: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-03-18T16:02:11.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:11 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:11.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:11 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:11.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:11 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:11 smithi191 ceph-mon[36784]: pgmap v66: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-03-18T16:02:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:11 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:11 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:11 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:11 smithi156 ceph-mon[35919]: pgmap v66: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.9 MiB/s wr, 330 op/s 2024-03-18T16:02:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:11 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:12.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:11 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:11 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 102709 10.02 MB/sec warmup 57 sec latency 17.573 ms 2024-03-18T16:02:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:12.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:12.843 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:12.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:12.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:12.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:13.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:13.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:13.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:02:13.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 104116 9.96 MB/sec warmup 58 sec latency 46.674 ms 2024-03-18T16:02:13.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:13 smithi150 ceph-mon[31946]: pgmap v67: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:13 smithi191 ceph-mon[36784]: pgmap v67: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:13 smithi156 ceph-mon[35919]: pgmap v67: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:14.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 106719 10.07 MB/sec warmup 59 sec latency 23.657 ms 2024-03-18T16:02:15.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 109021 10.04 MB/sec warmup 60 sec latency 15.618 ms 2024-03-18T16:02:15.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:15 smithi150 ceph-mon[31946]: pgmap v68: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:15 smithi191 ceph-mon[36784]: pgmap v68: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:15 smithi156 ceph-mon[35919]: pgmap v68: 97 pgs: 97 active+clean; 69 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 213 op/s 2024-03-18T16:02:16.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 110924 10.08 MB/sec warmup 61 sec latency 15.467 ms 2024-03-18T16:02:17.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 113630 10.19 MB/sec warmup 62 sec latency 90.603 ms 2024-03-18T16:02:17.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:17 smithi150 ceph-mon[31946]: pgmap v69: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 324 op/s 2024-03-18T16:02:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:17 smithi191 ceph-mon[36784]: pgmap v69: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 324 op/s 2024-03-18T16:02:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:17 smithi156 ceph-mon[35919]: pgmap v69: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.7 MiB/s wr, 324 op/s 2024-03-18T16:02:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.194+0000 7f454c681640 1 -- 172.21.15.150:0/3762864252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 msgr2=0x7f454410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.194+0000 7f454c681640 1 --2- 172.21.15.150:0/3762864252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f454410d510 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f454002bc30 comp rx=0 tx=0).stop 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- 172.21.15.150:0/3762864252 shutdown_connections 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- 172.21.15.150:0/3762864252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f454410d510 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- 172.21.15.150:0/3762864252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- 172.21.15.150:0/3762864252 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 0x7f4544105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- 172.21.15.150:0/3762864252 >> 172.21.15.150:0/3762864252 conn(0x7f4544077540 msgr2=0x7f4544077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- 172.21.15.150:0/3762864252 shutdown_connections 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- 172.21.15.150:0/3762864252 wait complete. 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 Processor -- start 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- start start 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 0x7f454419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4544111bc0 con 0x7f45441012f0 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4544111a40 con 0x7f4544100930 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454c681640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4544111d40 con 0x7f4544105680 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f4549c1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454a41d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f4549c1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53680/0 (socket says 172.21.15.150:53680) 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f4549c1c640 1 -- 172.21.15.150:0/899683012 learned_addr learned my addr 172.21.15.150:0/899683012 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.195+0000 7f454ac1e640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 0x7f454419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 -- 172.21.15.150:0/899683012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 msgr2=0x7f454419c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 0x7f454419c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 -- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 msgr2=0x7f45441983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 -- 172.21.15.150:0/899683012 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454419ce70 con 0x7f4544100930 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f4549c1c640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454a41d640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544197e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f453b7fe640 1 -- 172.21.15.150:0/899683012 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c004b20 con 0x7f4544100930 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454419d160 con 0x7f4544100930 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f453b7fe640 1 -- 172.21.15.150:0/899683012 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f453c004cc0 con 0x7f4544100930 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f453b7fe640 1 -- 172.21.15.150:0/899683012 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c0107b0 con 0x7f4544100930 2024-03-18T16:02:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.196+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45441a49f0 con 0x7f4544100930 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.198+0000 7f453b7fe640 1 -- 172.21.15.150:0/899683012 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f453c005020 con 0x7f4544100930 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 Processor -- start 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 -- start start 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1a50 0x5645ba0ba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0bcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5645ba015a40 con 0x5645ba0ba820 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5645b9fbe920 con 0x5645ba0b1130 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454f26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5645ba011ae0 con 0x5645ba0b1a50 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454ac1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454ac1e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:47138/0 (socket says 172.21.15.150:47138) 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f4549c1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0bcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454a41d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1a50 0x5645ba0ba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454ac1e640 1 -- 172.21.15.150:0/992173012 learned_addr learned my addr 172.21.15.150:0/992173012 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f453b7fe640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 0x7f45300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.199+0000 7f454ac1e640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 0x7f45300a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 -- 172.21.15.150:0/992173012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1a50 msgr2=0x5645ba0ba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1a50 0x5645ba0ba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 -- 172.21.15.150:0/992173012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 msgr2=0x5645ba0b1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 -- 172.21.15.150:0/992173012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645b9fd6460 con 0x5645ba0ba820 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f454ac1e640 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f453b7fe640 1 -- 172.21.15.150:0/899683012 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f453c0cf5b0 con 0x7f4544100930 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4549c1c640 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0bcc10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f45340114c0 tx=0x7f4534011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61854a7365a20a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f454ac1e640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 0x7f45300a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4540005ec0 tx=0x7f454002bc30 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4538ff9640 1 -- 172.21.15.150:0/992173012 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534004030 con 0x5645ba0ba820 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4538ff9640 1 -- 172.21.15.150:0/992173012 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45340041d0 con 0x5645ba0ba820 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.200+0000 7f4538ff9640 1 -- 172.21.15.150:0/992173012 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453400f960 con 0x5645ba0ba820 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- 172.21.15.150:0/992173012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 msgr2=0x5645ba0bcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0bcc10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f45340114c0 tx=0x7f4534011980 comp rx=0 tx=0).stop 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- 172.21.15.150:0/992173012 shutdown_connections 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0bcc10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1a50 0x5645ba0ba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- 172.21.15.150:0/992173012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1130 0x5645ba0b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- 172.21.15.150:0/992173012 >> 172.21.15.150:0/992173012 conn(0x5645ba0aba50 msgr2=0x5645ba0abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- 172.21.15.150:0/992173012 shutdown_connections 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- 172.21.15.150:0/992173012 wait complete. 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- start start 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 Processor -- start 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5645ba0bd450 con 0x5645ba0b1130 2024-03-18T16:02:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5645b9fd1d70 con 0x5645ba0b1a50 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5645ba0bd5d0 con 0x5645ba0ba820 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.201+0000 7f454f26e740 10 client.0 _ll_register_callbacks cb 0x5645b9d29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f4549c1c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53700/0 (socket says 172.21.15.150:53700) 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 -- 172.21.15.150:0/2814231479 learned_addr learned my addr 172.21.15.150:0/2814231479 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454ac1e640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 -- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 msgr2=0x5645ba0b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 -- 172.21.15.150:0/2814231479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 msgr2=0x5645ba0b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 -- 172.21.15.150:0/2814231479 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645b9d2a210 con 0x5645ba0b1130 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f4549c1c640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454a41d640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f453c00c760 tx=0x7f453c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda207179a7ad92e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f452dffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c09c300 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f453c0ccf70 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45340041d0 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f453c0cd170 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454f26e740 10 client.15594 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5645b9df6620 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.202+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45340041d0 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f453c0cf0d0 con 0x5645ba0b1130 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 10 client.15594 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 20 client.15594 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba0ca9d0 0x5645ba0ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 4 client.15594 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:02:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645ba022f80 con 0x5645ba0ca9d0 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f4549c1c640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba0ca9d0 0x5645ba0ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f452dffb640 10 client.15594 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.203+0000 7f4549c1c640 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba0ca9d0 0x5645ba0ccdc0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c82d902a2df487f6 server_cookie=c115285ea3c95d5c in_seq=0 out_seq=0 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f452dffb640 1 -- 172.21.15.150:0/2814231479 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645ba022f80 con 0x5645ba0ca9d0 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f452dffb640 10 client.15594 handle_command_reply: tid=0 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 client.15594 shutdown 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 msgr2=0x5645ba0b7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f453c00c760 tx=0x7f453c00cf20 comp rx=0 tx=0).stop 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 shutdown_connections 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba0ca9d0 0x5645ba0ccdc0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f452dffb640 0 client.15594 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 --2- 172.21.15.150:0/2814231479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b7870 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.204+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 >> 172.21.15.150:0/2814231479 conn(0x5645ba0aba50 msgr2=0x5645ba0ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.205+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 shutdown_connections 2024-03-18T16:02:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.205+0000 7f454f26e740 1 -- 172.21.15.150:0/2814231479 wait complete. 2024-03-18T16:02:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.205+0000 7f454f26e740 20 client.15594 trim_cache size 0 max 16384 2024-03-18T16:02:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 Processor -- start 2024-03-18T16:02:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 -- start start 2024-03-18T16:02:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1130 0x5645ba0b9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f45441afba0 con 0x5645ba0ba820 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ad410 con 0x5645ba0b1a50 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454f26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ab260 con 0x5645ba0b1130 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454ac1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454ac1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53706/0 (socket says 172.21.15.150:53706) 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454ac1e640 1 -- 172.21.15.150:0/187088894 learned_addr learned my addr 172.21.15.150:0/187088894 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f454a41d640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1130 0x5645ba0b9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.220+0000 7f4549c1c640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 -- 172.21.15.150:0/187088894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1130 msgr2=0x5645ba0b9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1130 0x5645ba0b9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 -- 172.21.15.150:0/187088894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 msgr2=0x5645ba0c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 -- 172.21.15.150:0/187088894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ba0b3450 con 0x5645ba0ba820 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f4549c1c640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f454ac1e640 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4540036040 tx=0x7f454002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a6b73091f12a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f452ffff640 1 -- 172.21.15.150:0/187088894 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45400046c0 con 0x5645ba0ba820 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f452ffff640 1 -- 172.21.15.150:0/187088894 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4540054070 con 0x5645ba0ba820 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.221+0000 7f452ffff640 1 -- 172.21.15.150:0/187088894 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45400035f0 con 0x5645ba0ba820 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- 172.21.15.150:0/187088894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 msgr2=0x5645ba0d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4540036040 tx=0x7f454002e040 comp rx=0 tx=0).stop 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- 172.21.15.150:0/187088894 shutdown_connections 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0ba820 0x5645ba0d2680 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- 172.21.15.150:0/187088894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0b1130 0x5645ba0b9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- 172.21.15.150:0/187088894 >> 172.21.15.150:0/187088894 conn(0x5645ba0aba50 msgr2=0x5645ba0ad730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- 172.21.15.150:0/187088894 shutdown_connections 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- 172.21.15.150:0/187088894 wait complete. 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- start start 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 Processor -- start 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ab8d0 con 0x5645ba0b1130 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45441acd40 con 0x5645ba0b1a50 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f45441abfa0 con 0x5645ba0ba820 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454f26e740 10 client.0 _ll_register_callbacks cb 0x7f453c005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454a41d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454a41d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53716/0 (socket says 172.21.15.150:53716) 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454a41d640 1 -- 172.21.15.150:0/3213388205 learned_addr learned my addr 172.21.15.150:0/3213388205 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f454ac1e640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.222+0000 7f4549c1c640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 -- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 msgr2=0x5645ba0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 -- 172.21.15.150:0/3213388205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 msgr2=0x5645ba0b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 -- 172.21.15.150:0/3213388205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ba12fcb0 con 0x5645ba0b1130 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f4549c1c640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454ac1e640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454a41d640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f453c0978e0 tx=0x7f453c0a42a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10d6b0ecbad0643 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c09bc00 con 0x5645ba0b1130 2024-03-18T16:02:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f453c0ccf70 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4540054070 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f453c0b3020 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 10 client.15600 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5645ba130f00 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4540054070 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f453c09aac0 con 0x5645ba0b1130 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 10 client.15600 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 10 client.15600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 20 client.15600 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba131630 0x5645ba133a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 4 client.15600 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.223+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5645ba01d2d0 con 0x5645ba131630 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.224+0000 7f4549c1c640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba131630 0x5645ba133a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.224+0000 7f452dffb640 10 client.15600 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.224+0000 7f4549c1c640 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba131630 0x5645ba133a50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98784b8e21722aa2 server_cookie=a3178599ae2cdf5a in_seq=0 out_seq=0 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f452dffb640 1 -- 172.21.15.150:0/3213388205 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5645ba01d2d0 con 0x5645ba131630 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f452dffb640 10 client.15600 handle_command_reply: tid=0 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 client.15600 shutdown 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 msgr2=0x5645ba0b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f453c0978e0 tx=0x7f453c0a42a0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 shutdown_connections 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5645ba131630 0x5645ba133a50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645ba0ba820 0x5645ba0b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f452dffb640 0 client.15600 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5645ba0b1a50 0x5645ba0b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 --2- 172.21.15.150:0/3213388205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5645ba0b1130 0x5645ba0b4350 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 >> 172.21.15.150:0/3213388205 conn(0x5645ba0aba50 msgr2=0x5645ba0acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 shutdown_connections 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 1 -- 172.21.15.150:0/3213388205 wait complete. 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.231+0000 7f454f26e740 20 client.15600 trim_cache size 0 max 16384 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:02:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:02:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 msgr2=0x7f45300a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 0x7f45300a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4540005ec0 tx=0x7f454002bc30 comp rx=0 tx=0).stop 2024-03-18T16:02:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 msgr2=0x7f4544197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544197e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c00ef90 comp rx=0 tx=0).stop 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 shutdown_connections 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f45300a2a10 0x7f45300a4ed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4544105680 0x7f454419c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45441012f0 0x7f45441983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 --2- 172.21.15.150:0/899683012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4544100930 0x7f4544197e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 >> 172.21.15.150:0/899683012 conn(0x7f4544077540 msgr2=0x7f45440fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 shutdown_connections 2024-03-18T16:02:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:18.232+0000 7f454c681640 1 -- 172.21.15.150:0/899683012 wait complete. 2024-03-18T16:02:18.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 115238 10.13 MB/sec warmup 63 sec latency 87.582 ms 2024-03-18T16:02:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 117553 10.19 MB/sec warmup 64 sec latency 15.911 ms 2024-03-18T16:02:19.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:19 smithi150 ceph-mon[31946]: pgmap v70: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 228 op/s 2024-03-18T16:02:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:19 smithi191 ceph-mon[36784]: pgmap v70: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 228 op/s 2024-03-18T16:02:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:19 smithi156 ceph-mon[35919]: pgmap v70: 97 pgs: 97 active+clean; 74 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 228 op/s 2024-03-18T16:02:20.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 120265 10.23 MB/sec warmup 65 sec latency 18.906 ms 2024-03-18T16:02:21.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 122085 10.25 MB/sec warmup 66 sec latency 18.945 ms 2024-03-18T16:02:21.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:21 smithi150 ceph-mon[31946]: pgmap v71: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 342 op/s 2024-03-18T16:02:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:21 smithi191 ceph-mon[36784]: pgmap v71: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 342 op/s 2024-03-18T16:02:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:21 smithi156 ceph-mon[35919]: pgmap v71: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 342 op/s 2024-03-18T16:02:22.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 124553 10.30 MB/sec warmup 67 sec latency 16.470 ms 2024-03-18T16:02:23.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 126642 10.26 MB/sec warmup 68 sec latency 81.637 ms 2024-03-18T16:02:23.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:23 smithi150 ceph-mon[31946]: pgmap v72: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:23 smithi191 ceph-mon[36784]: pgmap v72: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:23 smithi156 ceph-mon[35919]: pgmap v72: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:24.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 128393 10.28 MB/sec warmup 69 sec latency 32.867 ms 2024-03-18T16:02:25.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 131283 10.38 MB/sec warmup 70 sec latency 11.823 ms 2024-03-18T16:02:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:25 smithi191 ceph-mon[36784]: pgmap v73: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:25 smithi156 ceph-mon[35919]: pgmap v73: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:25 smithi150 ceph-mon[31946]: pgmap v73: 97 pgs: 97 active+clean; 78 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:26.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 133102 10.34 MB/sec warmup 71 sec latency 16.082 ms 2024-03-18T16:02:27.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 135380 10.38 MB/sec warmup 72 sec latency 19.027 ms 2024-03-18T16:02:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:27 smithi191 ceph-mon[36784]: pgmap v74: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:02:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:27 smithi156 ceph-mon[35919]: pgmap v74: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:02:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:27 smithi150 ceph-mon[31946]: pgmap v74: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:02:28.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 137599 10.39 MB/sec warmup 73 sec latency 82.855 ms 2024-03-18T16:02:29.315 INFO:tasks.workunit.client.0.smithi150.stdout: 1 139311 10.37 MB/sec warmup 74 sec latency 45.556 ms 2024-03-18T16:02:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:29 smithi191 ceph-mon[36784]: pgmap v75: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:02:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:29 smithi156 ceph-mon[35919]: pgmap v75: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:02:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:29 smithi150 ceph-mon[31946]: pgmap v75: 97 pgs: 97 active+clean; 83 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:02:30.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 142151 10.47 MB/sec warmup 75 sec latency 19.949 ms 2024-03-18T16:02:31.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 144410 10.44 MB/sec warmup 76 sec latency 17.601 ms 2024-03-18T16:02:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 -- 172.21.15.150:0/1384808059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c104e30 msgr2=0x7f3e8c105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 --2- 172.21.15.150:0/1384808059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c104e30 0x7f3e8c105210 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00aa10 tx=0x7f3e7c02cf70 comp rx=0 tx=0).stop 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 -- 172.21.15.150:0/1384808059 shutdown_connections 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 --2- 172.21.15.150:0/1384808059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100b00 0x7f3e8c10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 --2- 172.21.15.150:0/1384808059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c100140 0x7f3e8c1005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 --2- 172.21.15.150:0/1384808059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c104e30 0x7f3e8c105210 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 -- 172.21.15.150:0/1384808059 >> 172.21.15.150:0/1384808059 conn(0x7f3e8c076870 msgr2=0x7f3e8c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 -- 172.21.15.150:0/1384808059 shutdown_connections 2024-03-18T16:02:31.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.323+0000 7f3e90852640 1 -- 172.21.15.150:0/1384808059 wait complete. 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 Processor -- start 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 -- start start 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100140 0x7f3e8c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c104e30 0x7f3e8c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10f940 con 0x7f3e8c100b00 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10f7c0 con 0x7f3e8c104e30 2024-03-18T16:02:31.327 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e90852640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c10fac0 con 0x7f3e8c100140 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34628/0 (socket says 172.21.15.150:34628) 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 -- 172.21.15.150:0/1666671505 learned_addr learned my addr 172.21.15.150:0/1666671505 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100140 0x7f3e8c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c104e30 0x7f3e8c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 -- 172.21.15.150:0/1666671505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100140 msgr2=0x7f3e8c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100140 0x7f3e8c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 -- 172.21.15.150:0/1666671505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c104e30 msgr2=0x7f3e8c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c104e30 0x7f3e8c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.324+0000 7f3e81d9b640 1 -- 172.21.15.150:0/1666671505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c19cf10 con 0x7f3e8c100b00 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400e9f0 tx=0x7f3e7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e83fff640 1 -- 172.21.15.150:0/1666671505 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7400cd80 con 0x7f3e8c100b00 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e83fff640 1 -- 172.21.15.150:0/1666671505 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e74010910 con 0x7f3e8c100b00 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e83fff640 1 -- 172.21.15.150:0/1666671505 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74018990 con 0x7f3e8c100b00 2024-03-18T16:02:31.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c19d200 con 0x7f3e8c100b00 2024-03-18T16:02:31.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.325+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c1a4a90 con 0x7f3e8c100b00 2024-03-18T16:02:31.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.327+0000 7f3e83fff640 1 -- 172.21.15.150:0/1666671505 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f3e74010ac0 con 0x7f3e8c100b00 2024-03-18T16:02:31.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.327+0000 7f3e83fff640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 0x7f3e780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 0x7f3e780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e83fff640 1 -- 172.21.15.150:0/1666671505 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3e740cf0b0 con 0x7f3e8c100b00 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 Processor -- start 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 -- start start 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 0x7f3e780a4ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00a340 tx=0x7f3e7c005fd0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfbcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55925df15a40 con 0x55925dfb1130 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55925debe920 con 0x55925dfba820 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e9343e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55925df11ae0 con 0x55925dfb1a50 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34638/0 (socket says 172.21.15.150:34638) 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8a59c640 1 -- 172.21.15.150:0/185722379 learned_addr learned my addr 172.21.15.150:0/185722379 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.328+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfbcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 -- 172.21.15.150:0/185722379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 msgr2=0x55925dfba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 -- 172.21.15.150:0/185722379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 msgr2=0x55925dfbcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfbcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 -- 172.21.15.150:0/185722379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925ded6460 con 0x55925dfb1130 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfbcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c077dc0 tx=0x7f3e7c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5243121619e25c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e80d99640 1 -- 172.21.15.150:0/185722379 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c045070 con 0x55925dfb1130 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e80d99640 1 -- 172.21.15.150:0/185722379 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c051070 con 0x55925dfb1130 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.329+0000 7f3e80d99640 1 -- 172.21.15.150:0/185722379 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c005460 con 0x55925dfb1130 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- 172.21.15.150:0/185722379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 msgr2=0x55925dfb1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c077dc0 tx=0x7f3e7c03f000 comp rx=0 tx=0).stop 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- 172.21.15.150:0/185722379 shutdown_connections 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfbcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 --2- 172.21.15.150:0/185722379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1130 0x55925dfb1510 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- 172.21.15.150:0/185722379 >> 172.21.15.150:0/185722379 conn(0x55925dfaba50 msgr2=0x55925dfabe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- 172.21.15.150:0/185722379 shutdown_connections 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- 172.21.15.150:0/185722379 wait complete. 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 -- start start 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.330+0000 7f3e9343e740 1 Processor -- start 2024-03-18T16:02:31.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1130 0x55925dfb78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfb8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55925dfbd3f0 con 0x55925dfb1a50 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55925ded1d70 con 0x55925dfba820 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55925dfbd570 con 0x55925dfb1130 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 10 client.0 _ll_register_callbacks cb 0x55925dc29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34642/0 (socket says 172.21.15.150:34642) 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 -- 172.21.15.150:0/983061100 learned_addr learned my addr 172.21.15.150:0/983061100 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1130 0x55925dfb78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfb8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 -- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1130 msgr2=0x55925dfb78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1130 0x55925dfb78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 -- 172.21.15.150:0/983061100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 msgr2=0x55925dfb8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfb8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 -- 172.21.15.150:0/983061100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925dc2a210 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3e74097d10 tx=0x7f3e7409fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26607cae77b2009b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e740b4060 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3e740afbe0 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7c051070 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3e740862b0 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 10 client.15612 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.331+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55925dcf6620 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c051070 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3e7409cb30 con 0x55925dfb1a50 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 10 client.15612 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 20 client.15612 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:31.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925dfca9f0 0x55925dfcce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 4 client.15612 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55925df22f80 con 0x55925dfca9f0 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925dfca9f0 0x55925dfcce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e71ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.332+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925dfca9f0 0x55925dfcce00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6b8a7e64f4e44bf server_cookie=627ec846d923a527 in_seq=0 out_seq=0 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e71ffb640 1 -- 172.21.15.150:0/983061100 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55925df22f80 con 0x55925dfca9f0 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e71ffb640 10 client.15612 handle_command_reply: tid=0 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 client.15612 shutdown 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 msgr2=0x55925dfb7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3e74097d10 tx=0x7f3e7409fa90 comp rx=0 tx=0).stop 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 shutdown_connections 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925dfca9f0 0x55925dfcce00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfba820 0x55925dfb8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfb1a50 0x55925dfb7df0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e71ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 --2- 172.21.15.150:0/983061100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1130 0x55925dfb78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 >> 172.21.15.150:0/983061100 conn(0x55925dfaba50 msgr2=0x55925dfad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 shutdown_connections 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.333+0000 7f3e9343e740 1 -- 172.21.15.150:0/983061100 wait complete. 2024-03-18T16:02:31.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.334+0000 7f3e9343e740 20 client.15612 trim_cache size 0 max 16384 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 Processor -- start 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 -- start start 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfc6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1afbb0 con 0x55925dfba820 2024-03-18T16:02:31.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1ad420 con 0x55925dfb1130 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e9343e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1ab270 con 0x55925dfb1a50 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfc6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34648/0 (socket says 172.21.15.150:34648) 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8ad9d640 1 -- 172.21.15.150:0/960566127 learned_addr learned my addr 172.21.15.150:0/960566127 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 -- 172.21.15.150:0/960566127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 msgr2=0x55925dfb9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 -- 172.21.15.150:0/960566127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 msgr2=0x55925dfd2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 -- 172.21.15.150:0/960566127 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925dfb3490 con 0x55925dfb1a50 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.350+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfc6190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e7409b5b0 tx=0x7f3e74018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c51f97ca29661d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e73fff640 1 -- 172.21.15.150:0/960566127 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e740b4040 con 0x55925dfb1a50 2024-03-18T16:02:31.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e73fff640 1 -- 172.21.15.150:0/960566127 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7409cb30 con 0x55925dfb1a50 2024-03-18T16:02:31.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e73fff640 1 -- 172.21.15.150:0/960566127 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74097600 con 0x55925dfb1a50 2024-03-18T16:02:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 -- 172.21.15.150:0/960566127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 msgr2=0x55925dfc6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfc6190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e7409b5b0 tx=0x7f3e74018420 comp rx=0 tx=0).stop 2024-03-18T16:02:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 -- 172.21.15.150:0/960566127 shutdown_connections 2024-03-18T16:02:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfd2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfc6190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 --2- 172.21.15.150:0/960566127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 -- 172.21.15.150:0/960566127 >> 172.21.15.150:0/960566127 conn(0x55925dfaba50 msgr2=0x55925dfad730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 -- 172.21.15.150:0/960566127 shutdown_connections 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.351+0000 7f3e9343e740 1 -- 172.21.15.150:0/960566127 wait complete. 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 -- start start 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 Processor -- start 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfb4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1ab8e0 con 0x55925dfba820 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1acd50 con 0x55925dfb1130 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1abfb0 con 0x55925dfb1a50 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e9343e740 10 client.0 _ll_register_callbacks cb 0x7f3e7409b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:31.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34660/0 (socket says 172.21.15.150:34660) 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8ad9d640 1 -- 172.21.15.150:0/2894460651 learned_addr learned my addr 172.21.15.150:0/2894460651 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfb4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 -- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 msgr2=0x55925dfb4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfb4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 -- 172.21.15.150:0/2894460651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 msgr2=0x55925dfb5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8a59c640 1 -- 172.21.15.150:0/2894460651 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925e02ffe0 con 0x55925dfb1130 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.352+0000 7f3e8ad9d640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e8a59c640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb4780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00aa10 tx=0x7f3e7c00a370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7810e111d80f089b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c045040 con 0x55925dfb1130 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e7409cb30 con 0x55925dfb1130 2024-03-18T16:02:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f3e7c004020 con 0x55925dfb1130 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3e7c063020 con 0x55925dfb1130 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 10 client.25391 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55925e0312d0 con 0x55925dfb1130 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e7409cb30 con 0x55925dfb1130 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3e7c00b040 con 0x55925dfb1130 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 10 client.25391 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 10 client.25391 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 20 client.25391 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925e031610 0x55925e033a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 4 client.25391 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.353+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55925df1d2d0 con 0x55925e031610 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.354+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925e031610 0x55925e033a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.354+0000 7f3e71ffb640 10 client.25391 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:31.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.354+0000 7f3e81d9b640 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925e031610 0x55925e033a00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ae9d1872dd0adfe server_cookie=d400803089c14978 in_seq=0 out_seq=0 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e71ffb640 1 -- 172.21.15.150:0/2894460651 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55925df1d2d0 con 0x55925e031610 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e71ffb640 10 client.25391 handle_command_reply: tid=0 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 client.25391 shutdown 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 msgr2=0x55925dfb4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb4780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00aa10 tx=0x7f3e7c00a370 comp rx=0 tx=0).stop 2024-03-18T16:02:31.363 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 shutdown_connections 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55925e031610 0x55925e033a00 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55925dfba820 0x55925dfb5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e71ffb640 0 client.25391 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55925dfb1a50 0x55925dfb4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 --2- 172.21.15.150:0/2894460651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55925dfb1130 0x55925dfb4780 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 >> 172.21.15.150:0/2894460651 conn(0x55925dfaba50 msgr2=0x55925dfaa320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 shutdown_connections 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.361+0000 7f3e9343e740 1 -- 172.21.15.150:0/2894460651 wait complete. 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e9343e740 20 client.25391 trim_cache size 0 max 16384 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:02:31.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:02:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:02:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 msgr2=0x7f3e780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 0x7f3e780a4ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00a340 tx=0x7f3e7c005fd0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 msgr2=0x7f3e8c1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400e9f0 tx=0x7f3e7400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 shutdown_connections 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3e780a2a10 0x7f3e780a4ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e8c104e30 0x7f3e8c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e8c100b00 0x7f3e8c1984b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 --2- 172.21.15.150:0/1666671505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e8c100140 0x7f3e8c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 >> 172.21.15.150:0/1666671505 conn(0x7f3e8c076870 msgr2=0x7f3e8c102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 shutdown_connections 2024-03-18T16:02:31.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:31.362+0000 7f3e90852640 1 -- 172.21.15.150:0/1666671505 wait complete. 2024-03-18T16:02:31.892 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:31 smithi150 ceph-mon[31946]: pgmap v76: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-18T16:02:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:31 smithi191 ceph-mon[36784]: pgmap v76: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-18T16:02:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:31 smithi156 ceph-mon[35919]: pgmap v76: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-18T16:02:32.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 146346 10.46 MB/sec warmup 77 sec latency 25.444 ms 2024-03-18T16:02:33.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 148830 10.51 MB/sec warmup 78 sec latency 58.135 ms 2024-03-18T16:02:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:33 smithi191 ceph-mon[36784]: pgmap v77: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:33 smithi156 ceph-mon[35919]: pgmap v77: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:33 smithi150 ceph-mon[31946]: pgmap v77: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:34.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 150695 10.50 MB/sec warmup 79 sec latency 16.797 ms 2024-03-18T16:02:35.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 153093 10.54 MB/sec warmup 80 sec latency 9.726 ms 2024-03-18T16:02:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:35 smithi191 ceph-mon[36784]: pgmap v78: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:35 smithi156 ceph-mon[35919]: pgmap v78: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:35 smithi150 ceph-mon[31946]: pgmap v78: 97 pgs: 97 active+clean; 88 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-18T16:02:36.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 155569 10.57 MB/sec warmup 81 sec latency 13.520 ms 2024-03-18T16:02:37.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 157373 10.54 MB/sec warmup 82 sec latency 19.579 ms 2024-03-18T16:02:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:37 smithi191 ceph-mon[36784]: pgmap v79: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 335 op/s 2024-03-18T16:02:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:37 smithi156 ceph-mon[35919]: pgmap v79: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 335 op/s 2024-03-18T16:02:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:37 smithi150 ceph-mon[31946]: pgmap v79: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 335 op/s 2024-03-18T16:02:38.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 159878 10.62 MB/sec warmup 83 sec latency 99.273 ms 2024-03-18T16:02:39.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 161986 10.58 MB/sec warmup 84 sec latency 105.759 ms 2024-03-18T16:02:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:39 smithi191 ceph-mon[36784]: pgmap v80: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 222 op/s 2024-03-18T16:02:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:39 smithi156 ceph-mon[35919]: pgmap v80: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 222 op/s 2024-03-18T16:02:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:39 smithi150 ceph-mon[31946]: pgmap v80: 97 pgs: 97 active+clean; 93 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 222 op/s 2024-03-18T16:02:40.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 163905 10.60 MB/sec warmup 85 sec latency 13.195 ms 2024-03-18T16:02:41.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 166822 10.68 MB/sec warmup 86 sec latency 10.048 ms 2024-03-18T16:02:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:41 smithi191 ceph-mon[36784]: pgmap v81: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 333 op/s 2024-03-18T16:02:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:41 smithi156 ceph-mon[35919]: pgmap v81: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 333 op/s 2024-03-18T16:02:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:41 smithi150 ceph-mon[31946]: pgmap v81: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 333 op/s 2024-03-18T16:02:42.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 169079 10.65 MB/sec warmup 87 sec latency 19.536 ms 2024-03-18T16:02:43.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 170971 10.67 MB/sec warmup 88 sec latency 29.989 ms 2024-03-18T16:02:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:43 smithi191 ceph-mon[36784]: pgmap v82: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:43 smithi156 ceph-mon[35919]: pgmap v82: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:43 smithi150 ceph-mon[31946]: pgmap v82: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:44.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 173265 10.69 MB/sec warmup 89 sec latency 92.233 ms 2024-03-18T16:02:44.456 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 -- 172.21.15.150:0/3116636368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 msgr2=0x7fda84101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.456 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 --2- 172.21.15.150:0/3116636368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda84101c00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fda78009930 tx=0x7fda7802bc30 comp rx=0 tx=0).stop 2024-03-18T16:02:44.456 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 -- 172.21.15.150:0/3116636368 shutdown_connections 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 --2- 172.21.15.150:0/3116636368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84102140 0x7fda8410e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 --2- 172.21.15.150:0/3116636368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda84101c00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 --2- 172.21.15.150:0/3116636368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84107790 0x7fda84107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 -- 172.21.15.150:0/3116636368 >> 172.21.15.150:0/3116636368 conn(0x7fda840fd270 msgr2=0x7fda840ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 -- 172.21.15.150:0/3116636368 shutdown_connections 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.453+0000 7fda8a937640 1 -- 172.21.15.150:0/3116636368 wait complete. 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 Processor -- start 2024-03-18T16:02:44.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 -- start start 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84102140 0x7fda841985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 0x7fda8419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda84111ba0 con 0x7fda84101780 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda84111a20 con 0x7fda84107790 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda8a937640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fda84111d20 con 0x7fda84102140 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.454+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53370/0 (socket says 172.21.15.150:53370) 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 -- 172.21.15.150:0/1445940074 learned_addr learned my addr 172.21.15.150:0/1445940074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda837fe640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84102140 0x7fda841985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda88ed4640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 0x7fda8419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 -- 172.21.15.150:0/1445940074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84102140 msgr2=0x7fda841985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84102140 0x7fda841985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 -- 172.21.15.150:0/1445940074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 msgr2=0x7fda8419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 0x7fda8419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 -- 172.21.15.150:0/1445940074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8419d090 con 0x7fda84101780 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda88ed4640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 0x7fda8419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda83fff640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e9f0 tx=0x7fda7400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda817fa640 1 -- 172.21.15.150:0/1445940074 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7400cd80 con 0x7fda84101780 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda817fa640 1 -- 172.21.15.150:0/1445940074 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda74005500 con 0x7fda84101780 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8419d380 con 0x7fda84101780 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda817fa640 1 -- 172.21.15.150:0/1445940074 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74010690 con 0x7fda84101780 2024-03-18T16:02:44.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.455+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda841a4c10 con 0x7fda84101780 2024-03-18T16:02:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.457+0000 7fda817fa640 1 -- 172.21.15.150:0/1445940074 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fda740046c0 con 0x7fda84101780 2024-03-18T16:02:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda817fa640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 0x7fda6c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 Processor -- start 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 -- start start 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda817fa640 1 -- 172.21.15.150:0/1445940074 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fda740cf1a0 con 0x7fda84101780 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda837fe640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 0x7fda6c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b96c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556f70aefa40 con 0x556f70b8b130 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556f70a98920 con 0x556f70b94820 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556f70aebae0 con 0x556f70b8ba50 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53380/0 (socket says 172.21.15.150:53380) 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda83fff640 1 -- 172.21.15.150:0/776466064 learned_addr learned my addr 172.21.15.150:0/776466064 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda88ed4640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b96c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.458+0000 7fda837fe640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda837fe640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 0x7fda6c0a4ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda78005ab0 tx=0x7fda7802c5d0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 -- 172.21.15.150:0/776466064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 msgr2=0x556f70b942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 -- 172.21.15.150:0/776466064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 msgr2=0x556f70b96c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b96c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 -- 172.21.15.150:0/776466064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f70ab0460 con 0x556f70b8b130 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda88ed4640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda83fff640 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fda740a3ac0 tx=0x7fda7409b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bb8cf3e357ef66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda6a7fc640 1 -- 172.21.15.150:0/776466064 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7409b850 con 0x556f70b8b130 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda6a7fc640 1 -- 172.21.15.150:0/776466064 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda7409f9a0 con 0x556f70b8b130 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.459+0000 7fda6a7fc640 1 -- 172.21.15.150:0/776466064 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74086210 con 0x556f70b8b130 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- 172.21.15.150:0/776466064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 msgr2=0x556f70b8b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fda740a3ac0 tx=0x7fda7409b390 comp rx=0 tx=0).stop 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- 172.21.15.150:0/776466064 shutdown_connections 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b96c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/776466064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b8b510 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- 172.21.15.150:0/776466064 >> 172.21.15.150:0/776466064 conn(0x556f70b85a50 msgr2=0x556f70b85e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- 172.21.15.150:0/776466064 shutdown_connections 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- 172.21.15.150:0/776466064 wait complete. 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 -- start start 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.460+0000 7fda8d5f7740 1 Processor -- start 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b91e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b92380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556f70b97450 con 0x556f70b8b130 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556f70aabd70 con 0x556f70b94820 2024-03-18T16:02:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x556f70b975d0 con 0x556f70b8ba50 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda8d5f7740 10 client.0 _ll_register_callbacks cb 0x556f70803a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53384/0 (socket says 172.21.15.150:53384) 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 -- 172.21.15.150:0/4093481938 learned_addr learned my addr 172.21.15.150:0/4093481938 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda837fe640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b91e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda88ed4640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b92380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 -- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 msgr2=0x556f70b91e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b91e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 -- 172.21.15.150:0/4093481938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 msgr2=0x556f70b92380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b92380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 -- 172.21.15.150:0/4093481938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f70804210 con 0x556f70b8b130 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda88ed4640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b92380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda83fff640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fda740cf4c0 tx=0x7fda740980f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56419580fe77995d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.461+0000 7fda637fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740b8070 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fda740a3d70 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda7409f9a0 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda740ac030 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 10 client.15630 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556f708d0620 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda7409f9a0 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fda7409aca0 con 0x556f70b8b130 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 10 client.15630 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 20 client.15630 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70ba48f0 0x556f70ba6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 4 client.15630 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f70afcf80 con 0x556f70ba48f0 2024-03-18T16:02:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.462+0000 7fda837fe640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70ba48f0 0x556f70ba6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.463+0000 7fda637fe640 10 client.15630 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:44.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.463+0000 7fda837fe640 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70ba48f0 0x556f70ba6ce0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a592b023ed147b server_cookie=60ffb67f644f6973 in_seq=0 out_seq=0 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda637fe640 1 -- 172.21.15.150:0/4093481938 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f70afcf80 con 0x556f70ba48f0 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda637fe640 10 client.15630 handle_command_reply: tid=0 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 client.15630 shutdown 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 msgr2=0x556f70b91900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fda740cf4c0 tx=0x7fda740980f0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 shutdown_connections 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70ba48f0 0x556f70ba6ce0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70b92380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70b91e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda637fe640 0 client.15630 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/4093481938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70b91900 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 >> 172.21.15.150:0/4093481938 conn(0x556f70b85a50 msgr2=0x556f70b877a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 shutdown_connections 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 1 -- 172.21.15.150:0/4093481938 wait complete. 2024-03-18T16:02:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.464+0000 7fda8d5f7740 20 client.15630 trim_cache size 0 max 16384 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 Processor -- start 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 -- start start 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b8b130 0x556f70b94030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b94820 0x556f70b8eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda841afd30 con 0x556f70b8ba50 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ad5a0 con 0x556f70b8b130 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.479+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ab3f0 con 0x556f70b94820 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53390/0 (socket says 172.21.15.150:53390) 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 -- 172.21.15.150:0/2996218714 learned_addr learned my addr 172.21.15.150:0/2996218714 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda88ed4640 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b94820 0x556f70b8eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda83fff640 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b8b130 0x556f70b94030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 -- 172.21.15.150:0/2996218714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b94820 msgr2=0x556f70b8eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b94820 0x556f70b8eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 -- 172.21.15.150:0/2996218714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b8b130 msgr2=0x556f70b94030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b8b130 0x556f70b94030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 -- 172.21.15.150:0/2996218714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f70b974a0 con 0x556f70b8ba50 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda837fe640 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fda78036040 tx=0x7fda78041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a08ebf71753f6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda69ffb640 1 -- 172.21.15.150:0/2996218714 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78055c30 con 0x556f70b8ba50 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda69ffb640 1 -- 172.21.15.150:0/2996218714 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda78004040 con 0x556f70b8ba50 2024-03-18T16:02:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.480+0000 7fda69ffb640 1 -- 172.21.15.150:0/2996218714 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78004cf0 con 0x556f70b8ba50 2024-03-18T16:02:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- 172.21.15.150:0/2996218714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 msgr2=0x556f70b9ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fda78036040 tx=0x7fda78041000 comp rx=0 tx=0).stop 2024-03-18T16:02:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- 172.21.15.150:0/2996218714 shutdown_connections 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b94820 0x556f70b8eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8ba50 0x556f70b9ff30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/2996218714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b8b130 0x556f70b94030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- 172.21.15.150:0/2996218714 >> 172.21.15.150:0/2996218714 conn(0x556f70b85a50 msgr2=0x556f70b9f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- 172.21.15.150:0/2996218714 shutdown_connections 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- 172.21.15.150:0/2996218714 wait complete. 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.481+0000 7fda8d5f7740 1 -- start start 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 Processor -- start 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70ba83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70c038d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda841aba60 con 0x556f70b8b130 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda841aced0 con 0x556f70b94820 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fda841ac130 con 0x556f70b8ba50 2024-03-18T16:02:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 10 client.0 _ll_register_callbacks cb 0x7fda7c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53392/0 (socket says 172.21.15.150:53392) 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 -- 172.21.15.150:0/3728754951 learned_addr learned my addr 172.21.15.150:0/3728754951 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda837fe640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70ba83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda88ed4640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70c038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 -- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 msgr2=0x556f70ba83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70ba83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 -- 172.21.15.150:0/3728754951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 msgr2=0x556f70c038d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70c038d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 -- 172.21.15.150:0/3728754951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f70c03f10 con 0x556f70b8b130 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda83fff640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fda7409b640 tx=0x7fda74098160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858e1f1454fe8c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda637fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7408a020 con 0x556f70b8b130 2024-03-18T16:02:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fda74086110 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.482+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda78004040 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda740b8040 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 10 client.15636 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556f70c0b290 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda78004040 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fda7409aca0 con 0x556f70b8b130 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 10 client.15636 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 20 client.15636 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70c0b7e0 0x556f70c0dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 4 client.15636 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f70af72d0 con 0x556f70c0b7e0 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.483+0000 7fda837fe640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70c0b7e0 0x556f70c0dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.484+0000 7fda637fe640 10 client.15636 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:44.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.484+0000 7fda837fe640 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70c0b7e0 0x556f70c0dc00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f6974e26b4d0907 server_cookie=4d15850c4b4eef22 in_seq=0 out_seq=0 2024-03-18T16:02:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.491+0000 7fda637fe640 1 -- 172.21.15.150:0/3728754951 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556f70af72d0 con 0x556f70c0b7e0 2024-03-18T16:02:44.494 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.491+0000 7fda637fe640 10 client.15636 handle_command_reply: tid=0 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.491+0000 7fda8d5f7740 1 client.15636 shutdown 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 msgr2=0x556f70ba7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fda7409b640 tx=0x7fda74098160 comp rx=0 tx=0).stop 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 shutdown_connections 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x556f70c0b7e0 0x556f70c0dc00 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f70b94820 0x556f70c038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556f70b8ba50 0x556f70ba83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda637fe640 0 client.15636 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 --2- 172.21.15.150:0/3728754951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556f70b8b130 0x556f70ba7eb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 >> 172.21.15.150:0/3728754951 conn(0x556f70b85a50 msgr2=0x556f70b84bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 shutdown_connections 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 1 -- 172.21.15.150:0/3728754951 wait complete. 2024-03-18T16:02:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8d5f7740 20 client.15636 trim_cache size 0 max 16384 2024-03-18T16:02:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:02:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:02:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:02:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 msgr2=0x7fda6c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 0x7fda6c0a4ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fda78005ab0 tx=0x7fda7802c5d0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 msgr2=0x7fda841980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.492+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e9f0 tx=0x7fda7400edf0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 shutdown_connections 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fda6c0a2a10 0x7fda6c0a4ed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda84107790 0x7fda8419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fda84102140 0x7fda841985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 --2- 172.21.15.150:0/1445940074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda84101780 0x7fda841980a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 >> 172.21.15.150:0/1445940074 conn(0x7fda840fd270 msgr2=0x7fda840fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 shutdown_connections 2024-03-18T16:02:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:44.493+0000 7fda8a937640 1 -- 172.21.15.150:0/1445940074 wait complete. 2024-03-18T16:02:45.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 175211 10.70 MB/sec warmup 90 sec latency 15.514 ms 2024-03-18T16:02:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:45 smithi191 ceph-mon[36784]: pgmap v83: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:45 smithi156 ceph-mon[35919]: pgmap v83: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:45 smithi150 ceph-mon[31946]: pgmap v83: 97 pgs: 97 active+clean; 98 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 220 op/s 2024-03-18T16:02:46.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 177690 10.73 MB/sec warmup 91 sec latency 19.236 ms 2024-03-18T16:02:47.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 179679 10.69 MB/sec warmup 92 sec latency 199.433 ms 2024-03-18T16:02:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:47 smithi191 ceph-mon[36784]: pgmap v84: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 332 op/s 2024-03-18T16:02:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:47 smithi156 ceph-mon[35919]: pgmap v84: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 332 op/s 2024-03-18T16:02:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:47 smithi150 ceph-mon[31946]: pgmap v84: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 332 op/s 2024-03-18T16:02:48.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 181574 10.71 MB/sec warmup 93 sec latency 12.157 ms 2024-03-18T16:02:49.316 INFO:tasks.workunit.client.0.smithi150.stdout: 1 183886 10.73 MB/sec warmup 94 sec latency 115.053 ms 2024-03-18T16:02:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:49 smithi191 ceph-mon[36784]: pgmap v85: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-18T16:02:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:49 smithi156 ceph-mon[35919]: pgmap v85: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-18T16:02:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:49 smithi150 ceph-mon[31946]: pgmap v85: 97 pgs: 97 active+clean; 102 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-18T16:02:50.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 185559 10.71 MB/sec warmup 95 sec latency 19.531 ms 2024-03-18T16:02:51.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 188218 10.77 MB/sec warmup 96 sec latency 14.336 ms 2024-03-18T16:02:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:51 smithi191 ceph-mon[36784]: pgmap v86: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-03-18T16:02:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:51 smithi156 ceph-mon[35919]: pgmap v86: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-03-18T16:02:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:51 smithi150 ceph-mon[31946]: pgmap v86: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 326 op/s 2024-03-18T16:02:52.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 189923 10.72 MB/sec warmup 97 sec latency 95.146 ms 2024-03-18T16:02:53.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 192104 10.75 MB/sec warmup 98 sec latency 10.014 ms 2024-03-18T16:02:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:53 smithi191 ceph-mon[36784]: pgmap v87: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:53 smithi156 ceph-mon[35919]: pgmap v87: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:53 smithi150 ceph-mon[31946]: pgmap v87: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:54.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 194467 10.77 MB/sec warmup 99 sec latency 60.105 ms 2024-03-18T16:02:55.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 196189 10.75 MB/sec warmup 100 sec latency 22.315 ms 2024-03-18T16:02:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:55 smithi191 ceph-mon[36784]: pgmap v88: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:55 smithi156 ceph-mon[35919]: pgmap v88: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:55 smithi150 ceph-mon[31946]: pgmap v88: 97 pgs: 97 active+clean; 107 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 215 op/s 2024-03-18T16:02:56.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 198983 10.81 MB/sec warmup 101 sec latency 9.874 ms 2024-03-18T16:02:57.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 201515 10.81 MB/sec warmup 102 sec latency 18.221 ms 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 -- 172.21.15.150:0/710998886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 msgr2=0x7f19f810e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/710998886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f810e650 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009c10 tx=0x7f19f002bf00 comp rx=0 tx=0).stop 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 -- 172.21.15.150:0/710998886 shutdown_connections 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/710998886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f810e650 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/710998886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8101780 0x7f19f8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/710998886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19f8107790 0x7f19f8107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 -- 172.21.15.150:0/710998886 >> 172.21.15.150:0/710998886 conn(0x7f19f80fd2b0 msgr2=0x7f19f80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 -- 172.21.15.150:0/710998886 shutdown_connections 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.589+0000 7f19ff8e8640 1 -- 172.21.15.150:0/710998886 wait complete. 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 Processor -- start 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 -- start start 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19f8101780 0x7f19f8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8107790 0x7f19f819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8111bc0 con 0x7f19f8102140 2024-03-18T16:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8111a40 con 0x7f19f8107790 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19ff8e8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8111d40 con 0x7f19f8101780 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19fce83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19fce83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44162/0 (socket says 172.21.15.150:44162) 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19fce83640 1 -- 172.21.15.150:0/3226475442 learned_addr learned my addr 172.21.15.150:0/3226475442 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.590+0000 7f19fde85640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8107790 0x7f19f819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 -- 172.21.15.150:0/3226475442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19f8101780 msgr2=0x7f19f8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19f8101780 0x7f19f8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 -- 172.21.15.150:0/3226475442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8107790 msgr2=0x7f19f819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8107790 0x7f19f819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 -- 172.21.15.150:0/3226475442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f819d070 con 0x7f19f8102140 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19fce83640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f19e800b450 tx=0x7f19e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19ee7fc640 1 -- 172.21.15.150:0/3226475442 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e8015020 con 0x7f19f8102140 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f819d360 con 0x7f19f8102140 2024-03-18T16:02:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19ee7fc640 1 -- 172.21.15.150:0/3226475442 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e8011d70 con 0x7f19f8102140 2024-03-18T16:02:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19ee7fc640 1 -- 172.21.15.150:0/3226475442 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e800f340 con 0x7f19f8102140 2024-03-18T16:02:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.591+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f81a4bf0 con 0x7f19f8102140 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.595+0000 7f1a025af740 1 Processor -- start 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.595+0000 7f1a025af740 1 -- start start 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.595+0000 7f1a025af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f1a025af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f1a025af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a53c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f1a025af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5638389aca40 con 0x563838a48130 2024-03-18T16:02:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f1a025af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563838955920 con 0x563838a48a50 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f1a025af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5638389a8ae0 con 0x563838a51820 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44176/0 (socket says 172.21.15.150:44176) 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 -- 172.21.15.150:0/2036085179 learned_addr learned my addr 172.21.15.150:0/2036085179 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19ee7fc640 1 -- 172.21.15.150:0/3226475442 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f19e8018020 con 0x7f19f8102140 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fce83640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a53c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fd684640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 -- 172.21.15.150:0/2036085179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 msgr2=0x563838a53c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a53c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 -- 172.21.15.150:0/2036085179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 msgr2=0x563838a512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fde85640 1 -- 172.21.15.150:0/2036085179 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383896d460 con 0x563838a48130 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.596+0000 7f19fd684640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:02:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19ee7fc640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 0x7f19e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19fde85640 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009be0 tx=0x7f19f00098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb49357c998b3a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19ee7fc640 1 -- 172.21.15.150:0/3226475442 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f19e80ce7b0 con 0x7f19f8102140 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19e37fe640 1 -- 172.21.15.150:0/2036085179 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0033de0 con 0x563838a48130 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19e37fe640 1 -- 172.21.15.150:0/2036085179 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19f0003e50 con 0x563838a48130 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.597+0000 7f19e37fe640 1 -- 172.21.15.150:0/2036085179 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f002cac0 con 0x563838a48130 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f19fd684640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 0x7f19e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 -- 172.21.15.150:0/2036085179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 msgr2=0x563838a48510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f19f0009be0 tx=0x7f19f00098f0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 -- 172.21.15.150:0/2036085179 shutdown_connections 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a53c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 --2- 172.21.15.150:0/2036085179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838a48510 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f1a025af740 1 -- 172.21.15.150:0/2036085179 >> 172.21.15.150:0/2036085179 conn(0x563838a42a50 msgr2=0x563838a42e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.598+0000 7f19fd684640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 0x7f19e40a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19f4005ab0 tx=0x7f19f4005a40 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.599+0000 7f1a025af740 1 -- 172.21.15.150:0/2036085179 shutdown_connections 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.599+0000 7f1a025af740 1 -- 172.21.15.150:0/2036085179 wait complete. 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.599+0000 7f1a025af740 1 -- start start 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.599+0000 7f1a025af740 1 Processor -- start 2024-03-18T16:02:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a4edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a51820 0x563838a4f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563838a543f0 con 0x563838a51820 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563838968d70 con 0x563838a48a50 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.600+0000 7f1a025af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563838a54570 con 0x563838a48130 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33050/0 (socket says 172.21.15.150:33050) 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 -- 172.21.15.150:0/298014212 learned_addr learned my addr 172.21.15.150:0/298014212 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f1a025af740 10 client.0 _ll_register_callbacks cb 0x5638386c0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fce83640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a4edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fde85640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a51820 0x563838a4f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 -- 172.21.15.150:0/298014212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 msgr2=0x563838a4edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a4edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 -- 172.21.15.150:0/298014212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a51820 msgr2=0x563838a4f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a51820 0x563838a4f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.601+0000 7f19fd684640 1 -- 172.21.15.150:0/298014212 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638386c1210 con 0x563838a48130 2024-03-18T16:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.602+0000 7f19fd684640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009fd0 tx=0x7f19f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c145a52281a5ba36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.602+0000 7f19e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.602+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4023070 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.602+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f19f401f860 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.602+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19f0003e50 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.603+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f19f402f050 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.603+0000 7f1a025af740 10 client.25471 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.603+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56383878d620 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.603+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19f4012450 con 0x563838a48130 2024-03-18T16:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.603+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f19f40273b0 con 0x563838a48130 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 10 client.25471 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 10 client.25471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 20 client.25471 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838a63e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 4 client.25471 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f19f0003e50 con 0x563838a61a50 2024-03-18T16:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.604+0000 7f19fce83640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838a63e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.605+0000 7f19fce83640 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838a63e80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4aa681e8c8d0e89 server_cookie=63e3f181847ac2a4 in_seq=0 out_seq=0 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.605+0000 7f19e0ff9640 10 client.25471 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f19e0ff9640 1 -- 172.21.15.150:0/298014212 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f19f0003e50 con 0x563838a61a50 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f19e0ff9640 10 client.25471 handle_command_reply: tid=0 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 client.25471 shutdown 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 msgr2=0x563838a4e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009fd0 tx=0x7f19f40046c0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 shutdown_connections 2024-03-18T16:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838a63e80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a51820 0x563838a4f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f19e0ff9640 0 client.25471 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838a4edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 --2- 172.21.15.150:0/298014212 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a48130 0x563838a4e8b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.606+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 >> 172.21.15.150:0/298014212 conn(0x563838a42a50 msgr2=0x563838a447a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.607+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 shutdown_connections 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.607+0000 7f1a025af740 1 -- 172.21.15.150:0/298014212 wait complete. 2024-03-18T16:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.607+0000 7f1a025af740 20 client.25471 trim_cache size 0 max 16384 2024-03-18T16:02:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 Processor -- start 2024-03-18T16:02:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 -- start start 2024-03-18T16:02:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48130 0x563838a50fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a69790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81ada30 con 0x563838a48a50 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81ab2a0 con 0x563838a48130 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f1a025af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81a90f0 con 0x563838a51820 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f19fce83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f19fce83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44198/0 (socket says 172.21.15.150:44198) 2024-03-18T16:02:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f19fce83640 1 -- 172.21.15.150:0/4077926132 learned_addr learned my addr 172.21.15.150:0/4077926132 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.622+0000 7f19fde85640 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a69790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fd684640 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48130 0x563838a50fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 -- 172.21.15.150:0/4077926132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 msgr2=0x563838a69790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a69790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 -- 172.21.15.150:0/4077926132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48130 msgr2=0x563838a50fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48130 0x563838a50fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 -- 172.21.15.150:0/4077926132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563838a3e310 con 0x563838a48a50 2024-03-18T16:02:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19fce83640 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f19e800be10 tx=0x7f19e80bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90078e4b802ec27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19e2ffd640 1 -- 172.21.15.150:0/4077926132 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e809af70 con 0x563838a48a50 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19e2ffd640 1 -- 172.21.15.150:0/4077926132 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e809f0c0 con 0x563838a48a50 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.623+0000 7f19e2ffd640 1 -- 172.21.15.150:0/4077926132 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e80975e0 con 0x563838a48a50 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- 172.21.15.150:0/4077926132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 msgr2=0x563838a4f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f19e800be10 tx=0x7f19e80bb040 comp rx=0 tx=0).stop 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- 172.21.15.150:0/4077926132 shutdown_connections 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a69790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48a50 0x563838a4f150 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 --2- 172.21.15.150:0/4077926132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48130 0x563838a50fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- 172.21.15.150:0/4077926132 >> 172.21.15.150:0/4077926132 conn(0x563838a42a50 msgr2=0x563838a41320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- 172.21.15.150:0/4077926132 shutdown_connections 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- 172.21.15.150:0/4077926132 wait complete. 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 -- start start 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.624+0000 7f1a025af740 1 Processor -- start 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838ac1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a55870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81a9760 con 0x563838a48130 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81aabd0 con 0x563838a48a50 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f19f81a9e30 con 0x563838a51820 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44212/0 (socket says 172.21.15.150:44212) 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fce83640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838ac1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 -- 172.21.15.150:0/2746483597 learned_addr learned my addr 172.21.15.150:0/2746483597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fde85640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a55870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f1a025af740 10 client.0 _ll_register_callbacks cb 0x7f19f40099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 -- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 msgr2=0x563838a55870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a55870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 -- 172.21.15.150:0/2746483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 msgr2=0x563838ac1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838ac1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 -- 172.21.15.150:0/2746483597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563838a55fb0 con 0x563838a48130 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fce83640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838ac1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19fd684640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f19f40275e0 tx=0x7f19f4027610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666d4b0b0900037d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.625+0000 7f19e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:02:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4023050 con 0x563838a48130 2024-03-18T16:02:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f19f401f660 con 0x563838a48130 2024-03-18T16:02:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19e809f0c0 con 0x563838a48130 2024-03-18T16:02:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f19f40092a0 con 0x563838a48130 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 10 client.15654 fetch_fsmap learned FSMap version 25 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563838ad1060 con 0x563838a48130 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e809f0c0 con 0x563838a48130 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f19f4012a20 con 0x563838a48130 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 10 client.15654 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:02:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 20 client.15654 populate_metadata read hostname 'smithi150' 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838ad3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 4 client.15654 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638389b9f80 con 0x563838a61a50 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.628+0000 7f19fce83640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838ad3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.629+0000 7f19e0ff9640 10 client.15654 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.629+0000 7f19fce83640 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838ad3830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183781506737ed43 server_cookie=8352b4ca486df2a3 in_seq=0 out_seq=0 2024-03-18T16:02:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19e0ff9640 1 -- 172.21.15.150:0/2746483597 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638389b9f80 con 0x563838a61a50 2024-03-18T16:02:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19e0ff9640 10 client.15654 handle_command_reply: tid=0 2024-03-18T16:02:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 client.15654 shutdown 2024-03-18T16:02:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 msgr2=0x563838ac10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f19f40275e0 tx=0x7f19f4027610 comp rx=0 tx=0).stop 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 shutdown_connections 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563838a61a50 0x563838ad3830 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563838a51820 0x563838a55870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563838a48a50 0x563838ac1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 --2- 172.21.15.150:0/2746483597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563838a48130 0x563838ac10d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19e0ff9640 0 client.15654 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:02:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 >> 172.21.15.150:0/2746483597 conn(0x563838a42a50 msgr2=0x563838a41bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 shutdown_connections 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 1 -- 172.21.15.150:0/2746483597 wait complete. 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f1a025af740 20 client.15654 trim_cache size 0 max 16384 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:02:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:02:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:02:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 msgr2=0x7f19e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 0x7f19e40a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f19f4005ab0 tx=0x7f19f4005a40 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 msgr2=0x7f19f81985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.637+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f19e800b450 tx=0x7f19e800b910 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 shutdown_connections 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19e40a2a10 0x7f19e40a4ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f19f8107790 0x7f19f819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8102140 0x7f19f81985c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 --2- 172.21.15.150:0/3226475442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19f8101780 0x7f19f8198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 >> 172.21.15.150:0/3226475442 conn(0x7f19f80fd2b0 msgr2=0x7f19f80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 shutdown_connections 2024-03-18T16:02:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:02:57.638+0000 7f19ff8e8640 1 -- 172.21.15.150:0/3226475442 wait complete. 2024-03-18T16:02:57.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:57 smithi191 ceph-mon[36784]: pgmap v89: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-18T16:02:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:57 smithi156 ceph-mon[35919]: pgmap v89: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-18T16:02:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:57 smithi150 ceph-mon[31946]: pgmap v89: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-18T16:02:58.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 203235 10.80 MB/sec warmup 103 sec latency 28.238 ms 2024-03-18T16:02:59.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 205848 10.85 MB/sec warmup 104 sec latency 57.115 ms 2024-03-18T16:02:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:02:59 smithi191 ceph-mon[36784]: pgmap v90: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-18T16:03:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:02:59 smithi156 ceph-mon[35919]: pgmap v90: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-18T16:03:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:02:59 smithi150 ceph-mon[31946]: pgmap v90: 97 pgs: 97 active+clean; 111 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 209 op/s 2024-03-18T16:03:00.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 207555 10.81 MB/sec warmup 105 sec latency 19.932 ms 2024-03-18T16:03:01.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 209497 10.84 MB/sec warmup 106 sec latency 24.778 ms 2024-03-18T16:03:01.894 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:01 smithi150 ceph-mon[31946]: pgmap v91: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 339 op/s 2024-03-18T16:03:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:01 smithi191 ceph-mon[36784]: pgmap v91: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 339 op/s 2024-03-18T16:03:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:01 smithi156 ceph-mon[35919]: pgmap v91: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 339 op/s 2024-03-18T16:03:02.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 211638 10.80 MB/sec warmup 107 sec latency 18.699 ms 2024-03-18T16:03:03.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 213540 10.82 MB/sec warmup 108 sec latency 19.471 ms 2024-03-18T16:03:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:03 smithi191 ceph-mon[36784]: pgmap v92: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:03 smithi156 ceph-mon[35919]: pgmap v92: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:03 smithi150 ceph-mon[31946]: pgmap v92: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:04.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 215896 10.83 MB/sec warmup 109 sec latency 94.884 ms 2024-03-18T16:03:05.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 217824 10.84 MB/sec warmup 110 sec latency 14.113 ms 2024-03-18T16:03:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:05 smithi191 ceph-mon[36784]: pgmap v93: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:05 smithi156 ceph-mon[35919]: pgmap v93: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:05 smithi150 ceph-mon[31946]: pgmap v93: 97 pgs: 97 active+clean; 116 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 236 op/s 2024-03-18T16:03:06.317 INFO:tasks.workunit.client.0.smithi150.stdout: 1 220352 10.87 MB/sec warmup 111 sec latency 19.073 ms 2024-03-18T16:03:07.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 222864 10.88 MB/sec warmup 112 sec latency 17.816 ms 2024-03-18T16:03:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:07 smithi191 ceph-mon[36784]: pgmap v94: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 354 op/s 2024-03-18T16:03:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:07 smithi156 ceph-mon[35919]: pgmap v94: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 354 op/s 2024-03-18T16:03:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:07 smithi150 ceph-mon[31946]: pgmap v94: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 354 op/s 2024-03-18T16:03:08.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 224621 10.86 MB/sec warmup 113 sec latency 18.177 ms 2024-03-18T16:03:09.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 227057 10.91 MB/sec warmup 114 sec latency 81.699 ms 2024-03-18T16:03:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:09 smithi191 ceph-mon[36784]: pgmap v95: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-03-18T16:03:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:09 smithi156 ceph-mon[35919]: pgmap v95: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-03-18T16:03:10.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:09 smithi150 ceph-mon[31946]: pgmap v95: 97 pgs: 97 active+clean; 122 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 248 op/s 2024-03-18T16:03:10.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 228832 10.87 MB/sec warmup 115 sec latency 19.504 ms 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 -- 172.21.15.150:0/2228591864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c105680 msgr2=0x7f241c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 --2- 172.21.15.150:0/2228591864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c105680 0x7f241c105a60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f240c009900 tx=0x7f240c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 -- 172.21.15.150:0/2228591864 shutdown_connections 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 --2- 172.21.15.150:0/2228591864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c1012f0 0x7f241c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 --2- 172.21.15.150:0/2228591864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c100930 0x7f241c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 --2- 172.21.15.150:0/2228591864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c105680 0x7f241c105a60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 -- 172.21.15.150:0/2228591864 >> 172.21.15.150:0/2228591864 conn(0x7f241c077530 msgr2=0x7f241c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 -- 172.21.15.150:0/2228591864 shutdown_connections 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.729+0000 7f2422528640 1 -- 172.21.15.150:0/2228591864 wait complete. 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 Processor -- start 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 -- start start 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1012f0 0x7f241c1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c105680 0x7f241c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f241c110ba0 con 0x7f241c100930 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f241c110a20 con 0x7f241c1012f0 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2422528640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f241c110d20 con 0x7f241c105680 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f241bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f241bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60588/0 (socket says 172.21.15.150:60588) 2024-03-18T16:03:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f241bfff640 1 -- 172.21.15.150:0/1896005694 learned_addr learned my addr 172.21.15.150:0/1896005694 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f241b7fe640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1012f0 0x7f241c1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.730+0000 7f2420ac5640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c105680 0x7f241c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 -- 172.21.15.150:0/1896005694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c105680 msgr2=0x7f241c19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c105680 0x7f241c19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 -- 172.21.15.150:0/1896005694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1012f0 msgr2=0x7f241c1974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1012f0 0x7f241c1974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 -- 172.21.15.150:0/1896005694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c19bf20 con 0x7f241c100930 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f241bfff640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f240c005940 tx=0x7f240c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f24197fa640 1 -- 172.21.15.150:0/1896005694 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c041070 con 0x7f241c100930 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f24197fa640 1 -- 172.21.15.150:0/1896005694 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240c02ccc0 con 0x7f241c100930 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c19c1b0 con 0x7f241c100930 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f24197fa640 1 -- 172.21.15.150:0/1896005694 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c003d80 con 0x7f241c100930 2024-03-18T16:03:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.731+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c1a3aa0 con 0x7f241c100930 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.733+0000 7f24197fa640 1 -- 172.21.15.150:0/1896005694 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f240c044070 con 0x7f241c100930 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.733+0000 7f24251ea740 1 Processor -- start 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 -- start start 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132bb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 0x558c132bdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558c13216a40 con 0x558c132b2a50 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558c131bf920 con 0x558c132bb820 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24251ea740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558c13212ae0 con 0x558c132b2130 2024-03-18T16:03:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24197fa640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 0x7f24040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f2420ac5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f2420ac5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:36918/0 (socket says 172.21.15.150:36918) 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132bb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f2420ac5640 1 -- 172.21.15.150:0/3102515375 learned_addr learned my addr 172.21.15.150:0/3102515375 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 0x558c132bdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f2420ac5640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 0x7f24040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f24197fa640 1 -- 172.21.15.150:0/1896005694 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f240c0f07b0 con 0x7f241c100930 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 -- 172.21.15.150:0/3102515375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 msgr2=0x558c132b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 -- 172.21.15.150:0/3102515375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 msgr2=0x558c132bb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132bb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f241b7fe640 1 -- 172.21.15.150:0/3102515375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c131d7460 con 0x558c132bb820 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.734+0000 7f2420ac5640 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.735+0000 7f241b7fe640 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 0x558c132bdc10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2410005d00 tx=0x7f24100107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9e10e8a8582731a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.735+0000 7f2420ac5640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 0x7f24040a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2414009fd0 tx=0x7f2414009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.735+0000 7f24027fc640 1 -- 172.21.15.150:0/3102515375 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241001c480 con 0x558c132bb820 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.735+0000 7f24027fc640 1 -- 172.21.15.150:0/3102515375 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241001cb00 con 0x558c132bb820 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.735+0000 7f24027fc640 1 -- 172.21.15.150:0/3102515375 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410011b20 con 0x558c132bb820 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- 172.21.15.150:0/3102515375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 msgr2=0x558c132bdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 0x558c132bdc10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2410005d00 tx=0x7f24100107a0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- 172.21.15.150:0/3102515375 shutdown_connections 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132bb820 0x558c132bdc10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132bb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- 172.21.15.150:0/3102515375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132b2130 0x558c132b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- 172.21.15.150:0/3102515375 >> 172.21.15.150:0/3102515375 conn(0x558c132aca50 msgr2=0x558c132ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- 172.21.15.150:0/3102515375 shutdown_connections 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- 172.21.15.150:0/3102515375 wait complete. 2024-03-18T16:03:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- start start 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 Processor -- start 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.736+0000 7f24251ea740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558c132be450 con 0x558c132b2a50 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f24251ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558c131d2d70 con 0x558c132b2130 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f24251ea740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558c132be5d0 con 0x558c132bb820 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f24251ea740 10 client.0 _ll_register_callbacks cb 0x558c12f2aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60612/0 (socket says 172.21.15.150:60612) 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241b7fe640 1 -- 172.21.15.150:0/2213430067 learned_addr learned my addr 172.21.15.150:0/2213430067 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f2420ac5640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 -- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 msgr2=0x558c132b9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 -- 172.21.15.150:0/2213430067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 msgr2=0x558c132b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 -- 172.21.15.150:0/2213430067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c12f2b210 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f241bfff640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b87e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f240c0ee060 tx=0x7f240c046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ea00edca207317 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f23fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c04b040 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f240c0b9690 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.737+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f241001cb00 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f240c0a0050 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 10 client.25436 fetch_fsmap learned FSMap version 25 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c12ff7620 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241001cb00 con 0x558c132b2130 2024-03-18T16:03:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f240c0a8060 con 0x558c132b2130 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 10 client.25436 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 10 client.25436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 20 client.25436 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c132cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 4 client.25436 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c13223f80 con 0x558c132cb9e0 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.738+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c132cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c132cdea0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb39700b82f3130f server_cookie=d4b64d59c64ad4b2 in_seq=0 out_seq=0 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f23fb7fe640 10 client.25436 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2213430067 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c13223f80 con 0x558c132cb9e0 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f23fb7fe640 10 client.25436 handle_command_reply: tid=0 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 client.25436 shutdown 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 msgr2=0x558c132b87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b87e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f240c0ee060 tx=0x7f240c046000 comp rx=0 tx=0).stop 2024-03-18T16:03:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 shutdown_connections 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c132cdea0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f23fb7fe640 0 client.25436 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 --2- 172.21.15.150:0/2213430067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b87e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.739+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 >> 172.21.15.150:0/2213430067 conn(0x558c132aca50 msgr2=0x558c132ae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.740+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 shutdown_connections 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.740+0000 7f24251ea740 1 -- 172.21.15.150:0/2213430067 wait complete. 2024-03-18T16:03:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.740+0000 7f24251ea740 20 client.25436 trim_cache size 0 max 16384 2024-03-18T16:03:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 Processor -- start 2024-03-18T16:03:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 -- start start 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1aea50 con 0x558c132b2a50 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1ac2c0 con 0x558c132b2130 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f24251ea740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1aa110 con 0x558c132bb820 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f241bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60624/0 (socket says 172.21.15.150:60624) 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f241bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:44364/0 (socket says 172.21.15.150:44364) 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f241b7fe640 1 -- 172.21.15.150:0/1379697534 learned_addr learned my addr 172.21.15.150:0/1379697534 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.755+0000 7f2420ac5640 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 -- 172.21.15.150:0/1379697534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 msgr2=0x558c132c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 -- 172.21.15.150:0/1379697534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 msgr2=0x558c132c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 -- 172.21.15.150:0/1379697534 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c132b2880 con 0x558c132b2130 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241b7fe640 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f241bfff640 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f240c0ee550 tx=0x7f240c02ce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be28d5f1dd52ac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f2401ffb640 1 -- 172.21.15.150:0/1379697534 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c04b070 con 0x558c132b2130 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f2401ffb640 1 -- 172.21.15.150:0/1379697534 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240c0c10c0 con 0x558c132b2130 2024-03-18T16:03:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.756+0000 7f2401ffb640 1 -- 172.21.15.150:0/1379697534 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c0c5710 con 0x558c132b2130 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- 172.21.15.150:0/1379697534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 msgr2=0x558c132baf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f240c0ee550 tx=0x7f240c02ce70 comp rx=0 tx=0).stop 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- 172.21.15.150:0/1379697534 shutdown_connections 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- 172.21.15.150:0/1379697534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132baf10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- 172.21.15.150:0/1379697534 >> 172.21.15.150:0/1379697534 conn(0x558c132aca50 msgr2=0x558c132ab320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- 172.21.15.150:0/1379697534 shutdown_connections 2024-03-18T16:03:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- 172.21.15.150:0/1379697534 wait complete. 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- start start 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 Processor -- start 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1aa780 con 0x558c132b2a50 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1abbf0 con 0x558c132b2130 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f241c1aae50 con 0x558c132bb820 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.757+0000 7f24251ea740 10 client.0 _ll_register_callbacks cb 0x7f240c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60626/0 (socket says 172.21.15.150:60626) 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 -- 172.21.15.150:0/2000888257 learned_addr learned my addr 172.21.15.150:0/2000888257 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f2420ac5640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 -- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 msgr2=0x558c132b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 -- 172.21.15.150:0/2000888257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 msgr2=0x558c132b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 -- 172.21.15.150:0/2000888257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c13332df0 con 0x558c132b2a50 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241bfff640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f241b7fe640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2410006010 tx=0x7f2410011740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a545c726fb173cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f23fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241000be60 con 0x558c132b2a50 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f2410005450 con 0x558c132b2a50 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f240c0c10c0 con 0x558c132b2a50 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2410036020 con 0x558c132b2a50 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f24251ea740 10 client.15672 fetch_fsmap learned FSMap version 25 2024-03-18T16:03:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c13334130 con 0x558c132b2a50 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.758+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240c0c10c0 con 0x558c132b2a50 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f24100118d0 con 0x558c132b2a50 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 10 client.15672 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 20 client.15672 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c133366e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 4 client.15672 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c1321e2d0 con 0x558c132cb9e0 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f241bfff640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c133366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f23fb7fe640 10 client.15672 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.759+0000 7f241bfff640 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c133366e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3e1b89b00db0b8 server_cookie=1a265f6eb1edbf90 in_seq=0 out_seq=0 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f23fb7fe640 1 -- 172.21.15.150:0/2000888257 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c1321e2d0 con 0x558c132cb9e0 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f23fb7fe640 10 client.15672 handle_command_reply: tid=0 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 client.15672 shutdown 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 msgr2=0x558c132b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2410006010 tx=0x7f2410011740 comp rx=0 tx=0).stop 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 shutdown_connections 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558c132cb9e0 0x558c133366e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558c132bb820 0x558c132b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558c132b2a50 0x558c132b4f70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f23fb7fe640 0 client.15672 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 --2- 172.21.15.150:0/2000888257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c132b2130 0x558c132b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 >> 172.21.15.150:0/2000888257 conn(0x558c132aca50 msgr2=0x558c132ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 shutdown_connections 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 1 -- 172.21.15.150:0/2000888257 wait complete. 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.768+0000 7f24251ea740 20 client.15672 trim_cache size 0 max 16384 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:03:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:03:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 msgr2=0x7f24040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 0x7f24040a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2414009fd0 tx=0x7f2414009500 comp rx=0 tx=0).stop 2024-03-18T16:03:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 msgr2=0x7f241c196f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f240c005940 tx=0x7f240c0042f0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 shutdown_connections 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24040a2a10 0x7f24040a4ed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f241c105680 0x7f241c19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f241c1012f0 0x7f241c1974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 --2- 172.21.15.150:0/1896005694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f241c100930 0x7f241c196f80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 >> 172.21.15.150:0/1896005694 conn(0x7f241c077530 msgr2=0x7f241c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 shutdown_connections 2024-03-18T16:03:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:10.769+0000 7f2422528640 1 -- 172.21.15.150:0/1896005694 wait complete. 2024-03-18T16:03:11.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 231187 10.90 MB/sec warmup 116 sec latency 8.700 ms 2024-03-18T16:03:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:11 smithi191 ceph-mon[36784]: pgmap v96: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.4 MiB/s wr, 352 op/s 2024-03-18T16:03:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:11 smithi156 ceph-mon[35919]: pgmap v96: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.4 MiB/s wr, 352 op/s 2024-03-18T16:03:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:11 smithi150 ceph-mon[31946]: pgmap v96: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.4 MiB/s wr, 352 op/s 2024-03-18T16:03:12.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 233587 10.91 MB/sec warmup 117 sec latency 22.806 ms 2024-03-18T16:03:12.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:12 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:13.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:12 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:13.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:12 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:13.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 235255 10.89 MB/sec warmup 118 sec latency 20.239 ms 2024-03-18T16:03:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:13 smithi191 ceph-mon[36784]: pgmap v97: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:13 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:13 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:13 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:03:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:13 smithi156 ceph-mon[35919]: pgmap v97: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:13 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:13 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:13 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:03:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:13 smithi150 ceph-mon[31946]: pgmap v97: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:13 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:13 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:13 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:03:14.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 237764 10.93 MB/sec warmup 119 sec latency 57.441 ms 2024-03-18T16:03:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:15 smithi191 ceph-mon[36784]: pgmap v98: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:15 smithi156 ceph-mon[35919]: pgmap v98: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:15 smithi150 ceph-mon[31946]: pgmap v98: 97 pgs: 97 active+clean; 127 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-03-18T16:03:16.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 241783 13.87 MB/sec execute 1 sec latency 17.244 ms 2024-03-18T16:03:17.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 244445 13.59 MB/sec execute 2 sec latency 11.705 ms 2024-03-18T16:03:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:17 smithi191 ceph-mon[36784]: pgmap v99: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:03:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:17 smithi156 ceph-mon[35919]: pgmap v99: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:03:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:17 smithi150 ceph-mon[31946]: pgmap v99: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.4 MiB/s wr, 337 op/s 2024-03-18T16:03:18.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 246377 12.72 MB/sec execute 3 sec latency 19.322 ms 2024-03-18T16:03:18.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:18 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:18 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:19.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:18 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 248574 12.95 MB/sec execute 4 sec latency 78.005 ms 2024-03-18T16:03:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:19 smithi191 ceph-mon[36784]: pgmap v100: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-18T16:03:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:19 smithi156 ceph-mon[35919]: pgmap v100: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-18T16:03:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:19 smithi150 ceph-mon[31946]: pgmap v100: 97 pgs: 97 active+clean; 133 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-18T16:03:20.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 250718 11.82 MB/sec execute 5 sec latency 27.692 ms 2024-03-18T16:03:21.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 252546 11.89 MB/sec execute 6 sec latency 19.592 ms 2024-03-18T16:03:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:21 smithi191 ceph-mon[36784]: pgmap v101: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 330 op/s 2024-03-18T16:03:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:21 smithi156 ceph-mon[35919]: pgmap v101: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 330 op/s 2024-03-18T16:03:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:21 smithi150 ceph-mon[31946]: pgmap v101: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 330 op/s 2024-03-18T16:03:22.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 255241 12.24 MB/sec execute 7 sec latency 17.308 ms 2024-03-18T16:03:23.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 257030 11.94 MB/sec execute 8 sec latency 19.837 ms 2024-03-18T16:03:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 -- 172.21.15.150:0/2957824158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 msgr2=0x7fb30010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 --2- 172.21.15.150:0/2957824158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb30010d510 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009ca0 tx=0x7fb2f802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 -- 172.21.15.150:0/2957824158 shutdown_connections 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 --2- 172.21.15.150:0/2957824158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb30010d510 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 --2- 172.21.15.150:0/2957824158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300100930 0x7fb300100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 --2- 172.21.15.150:0/2957824158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300105680 0x7fb300105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 -- 172.21.15.150:0/2957824158 >> 172.21.15.150:0/2957824158 conn(0x7fb300077530 msgr2=0x7fb300077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 -- 172.21.15.150:0/2957824158 shutdown_connections 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.857+0000 7fb305177640 1 -- 172.21.15.150:0/2957824158 wait complete. 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 Processor -- start 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 -- start start 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300100930 0x7fb300198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300105680 0x7fb30019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb300110ba0 con 0x7fb3001012f0 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb300110a20 con 0x7fb300105680 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb305177640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb300110d20 con 0x7fb300100930 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2fed9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300100930 0x7fb300198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49788/0 (socket says 172.21.15.150:49788) 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 -- 172.21.15.150:0/3887662259 learned_addr learned my addr 172.21.15.150:0/3887662259 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300105680 0x7fb30019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 -- 172.21.15.150:0/3887662259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300100930 msgr2=0x7fb300198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300100930 0x7fb300198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 -- 172.21.15.150:0/3887662259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300105680 msgr2=0x7fb30019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300105680 0x7fb30019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 -- 172.21.15.150:0/3887662259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30019d020 con 0x7fb3001012f0 2024-03-18T16:03:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800ea60 tx=0x7fb2e800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.858+0000 7fb2fcd99640 1 -- 172.21.15.150:0/3887662259 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800cde0 con 0x7fb3001012f0 2024-03-18T16:03:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.859+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30019d310 con 0x7fb3001012f0 2024-03-18T16:03:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.859+0000 7fb2fcd99640 1 -- 172.21.15.150:0/3887662259 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2e80035f0 con 0x7fb3001012f0 2024-03-18T16:03:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.859+0000 7fb2fcd99640 1 -- 172.21.15.150:0/3887662259 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8010670 con 0x7fb3001012f0 2024-03-18T16:03:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.859+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3001a4ba0 con 0x7fb3001012f0 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.861+0000 7fb2fcd99640 1 -- 172.21.15.150:0/3887662259 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fb2e80037a0 con 0x7fb3001012f0 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.861+0000 7fb3079fd740 1 Processor -- start 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.861+0000 7fb3079fd740 1 -- start start 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d62453510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d6245c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560d623b7a40 con 0x560d62453130 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fcd99640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 0x7fb2ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560d62360920 con 0x560d62453a50 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb3079fd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560d623b3ae0 con 0x560d6245c820 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d62453510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d6245c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d62453510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49796/0 (socket says 172.21.15.150:49796) 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2ff59e640 1 -- 172.21.15.150:0/1118161328 learned_addr learned my addr 172.21.15.150:0/1118161328 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 0x7fb2ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fcd99640 1 -- 172.21.15.150:0/3887662259 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fb2e80cf100 con 0x7fb3001012f0 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 -- 172.21.15.150:0/1118161328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 msgr2=0x560d6245ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 -- 172.21.15.150:0/1118161328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 msgr2=0x560d62453510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d62453510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2fed9d640 1 -- 172.21.15.150:0/1118161328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d62378460 con 0x560d62453a50 2024-03-18T16:03:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.862+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 0x7fb2ec0a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8037040 tx=0x7fb2f8005c50 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d6245c2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009c00 tx=0x7fb2f0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc777aea1e735c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb2f57fa640 1 -- 172.21.15.150:0/1118161328 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0004230 con 0x560d62453a50 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb2f57fa640 1 -- 172.21.15.150:0/1118161328 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f00043d0 con 0x560d62453a50 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb2f57fa640 1 -- 172.21.15.150:0/1118161328 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0004d90 con 0x560d62453a50 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb3079fd740 1 -- 172.21.15.150:0/1118161328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 msgr2=0x560d6245c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.863+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d6245c2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009c00 tx=0x7fb2f0011810 comp rx=0 tx=0).stop 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- 172.21.15.150:0/1118161328 shutdown_connections 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d6245c2e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1118161328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d62453510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- 172.21.15.150:0/1118161328 >> 172.21.15.150:0/1118161328 conn(0x560d6244da50 msgr2=0x560d6244de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- 172.21.15.150:0/1118161328 shutdown_connections 2024-03-18T16:03:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- 172.21.15.150:0/1118161328 wait complete. 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- start start 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 Processor -- start 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453130 0x560d62459820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62459d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560d6245f450 con 0x560d6245c820 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560d62373d70 con 0x560d62453a50 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560d6245f5d0 con 0x560d62453130 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb3079fd740 10 client.0 _ll_register_callbacks cb 0x560d620cba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb2fed9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453130 0x560d62459820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49810/0 (socket says 172.21.15.150:49810) 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb2ff59e640 1 -- 172.21.15.150:0/1570754626 learned_addr learned my addr 172.21.15.150:0/1570754626 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.864+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62459d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 -- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453130 msgr2=0x560d62459820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453130 0x560d62459820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 -- 172.21.15.150:0/1570754626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 msgr2=0x560d62459d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62459d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 -- 172.21.15.150:0/1570754626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d620cc210 con 0x560d6245c820 2024-03-18T16:03:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb2f802c630 tx=0x7fb2f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8eed7b22d7eec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8052060 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fb2f8003970 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2f00043d0 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2f8039040 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 10 client.15684 fetch_fsmap learned FSMap version 27 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d62198620 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f00043d0 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb2f8042480 con 0x560d6245c820 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 10 client.15684 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 20 client.15684 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d6246c960 0x560d6246ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 4 client.15684 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.865+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d623c4f80 con 0x560d6246c960 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.866+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d6246c960 0x560d6246ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.866+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d6246c960 0x560d6246ed70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=254f013736762877 server_cookie=4f1c9d4102bf0f89 in_seq=0 out_seq=0 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.866+0000 7fb2e67fc640 10 client.15684 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb2e67fc640 1 -- 172.21.15.150:0/1570754626 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d623c4f80 con 0x560d6246c960 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb2e67fc640 10 client.15684 handle_command_reply: tid=0 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 client.15684 shutdown 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 msgr2=0x560d6245a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb2f802c630 tx=0x7fb2f80095c0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 shutdown_connections 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d6246c960 0x560d6246ed70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d6245a2a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62459d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 --2- 172.21.15.150:0/1570754626 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453130 0x560d62459820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb2e67fc640 0 client.15684 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 >> 172.21.15.150:0/1570754626 conn(0x560d6244da50 msgr2=0x560d6244f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 shutdown_connections 2024-03-18T16:03:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 1 -- 172.21.15.150:0/1570754626 wait complete. 2024-03-18T16:03:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.867+0000 7fb3079fd740 20 client.15684 trim_cache size 0 max 16384 2024-03-18T16:03:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.881+0000 7fb3079fd740 1 Processor -- start 2024-03-18T16:03:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 -- start start 2024-03-18T16:03:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62474680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001afc40 con 0x560d62453130 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001ad4e0 con 0x560d62453a50 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb3079fd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001ab330 con 0x560d6245c820 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49822/0 (socket says 172.21.15.150:49822) 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 -- 172.21.15.150:0/3151692135 learned_addr learned my addr 172.21.15.150:0/3151692135 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62474680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 -- 172.21.15.150:0/3151692135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 msgr2=0x560d6245af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 -- 172.21.15.150:0/3151692135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 msgr2=0x560d62474680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62474680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 -- 172.21.15.150:0/3151692135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d62452900 con 0x560d62453130 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.882+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x560d62452be0 tx=0x7fb2f0004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3deb7859672624 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb2f4ff9640 1 -- 172.21.15.150:0/3151692135 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0032070 con 0x560d62453130 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb2f4ff9640 1 -- 172.21.15.150:0/3151692135 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f002a070 con 0x560d62453130 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb2f4ff9640 1 -- 172.21.15.150:0/3151692135 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f001e040 con 0x560d62453130 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 -- 172.21.15.150:0/3151692135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 msgr2=0x560d624681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x560d62452be0 tx=0x7fb2f0004a10 comp rx=0 tx=0).stop 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 -- 172.21.15.150:0/3151692135 shutdown_connections 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d6245c820 0x560d6245af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453a50 0x560d62474680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3151692135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d62453130 0x560d624681f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 -- 172.21.15.150:0/3151692135 >> 172.21.15.150:0/3151692135 conn(0x560d6244da50 msgr2=0x560d6244de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 -- 172.21.15.150:0/3151692135 shutdown_connections 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.883+0000 7fb3079fd740 1 -- 172.21.15.150:0/3151692135 wait complete. 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 -- start start 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 Processor -- start 2024-03-18T16:03:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453130 0x560d624600d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453a50 0x560d62460610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001ab9a0 con 0x560d6245c820 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001ace10 con 0x560d62453130 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb3001ac070 con 0x560d62453a50 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb3079fd740 10 client.0 _ll_register_callbacks cb 0x7fb2e8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49834/0 (socket says 172.21.15.150:49834) 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 -- 172.21.15.150:0/3993055142 learned_addr learned my addr 172.21.15.150:0/3993055142 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453130 0x560d624600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2f7fff640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453a50 0x560d62460610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 -- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453a50 msgr2=0x560d62460610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453a50 0x560d62460610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 -- 172.21.15.150:0/3993055142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453130 msgr2=0x560d624600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453130 0x560d624600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 -- 172.21.15.150:0/3993055142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d624d1cb0 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2ff59e640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009590 tx=0x7fb2f80323a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b448f91f774672 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.884+0000 7fb2e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8052070 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fb2f8003970 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2f002a070 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2f8042640 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 10 client.15690 fetch_fsmap learned FSMap version 27 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d624d2f00 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f002a070 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) ==== 8346+0+0 (secure 0 0 0) 0x7fb2f8057460 con 0x560d6245c820 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 10 client.15690 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:03:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 20 client.15690 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d624d3530 0x560d624d5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 4 client.15690 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d623bf2d0 con 0x560d624d3530 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.885+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d624d3530 0x560d624d5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.886+0000 7fb2fed9d640 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d624d3530 0x560d624d5950 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2057a51e3306790 server_cookie=8f964d766b186695 in_seq=0 out_seq=0 2024-03-18T16:03:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.886+0000 7fb2e67fc640 10 client.15690 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb2e67fc640 1 -- 172.21.15.150:0/3993055142 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d623bf2d0 con 0x560d624d3530 2024-03-18T16:03:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb2e67fc640 10 client.15690 handle_command_reply: tid=0 2024-03-18T16:03:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 client.15690 shutdown 2024-03-18T16:03:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 msgr2=0x560d62460b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009590 tx=0x7fb2f80323a0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 shutdown_connections 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560d624d3530 0x560d624d5950 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560d6245c820 0x560d62460b50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb2e67fc640 0 client.15690 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560d62453a50 0x560d62460610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 --2- 172.21.15.150:0/3993055142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d62453130 0x560d624600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 >> 172.21.15.150:0/3993055142 conn(0x560d6244da50 msgr2=0x560d6244ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 shutdown_connections 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.894+0000 7fb3079fd740 1 -- 172.21.15.150:0/3993055142 wait complete. 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb3079fd740 20 client.15690 trim_cache size 0 max 16384 2024-03-18T16:03:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:03:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:03:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:03:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 msgr2=0x7fb2ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 0x7fb2ec0a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8037040 tx=0x7fb2f8005c50 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 msgr2=0x7fb300198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800ea60 tx=0x7fb2e800ef20 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 shutdown_connections 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb2ec0a2a10 0x7fb2ec0a4ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb300105680 0x7fb30019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3001012f0 0x7fb300198570 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 --2- 172.21.15.150:0/3887662259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb300100930 0x7fb300198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 >> 172.21.15.150:0/3887662259 conn(0x7fb300077530 msgr2=0x7fb300104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 shutdown_connections 2024-03-18T16:03:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:23.895+0000 7fb305177640 1 -- 172.21.15.150:0/3887662259 wait complete. 2024-03-18T16:03:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:23 smithi191 ceph-mon[36784]: pgmap v102: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:23 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:23 smithi156 ceph-mon[35919]: pgmap v102: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:23 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:23 smithi150 ceph-mon[31946]: pgmap v102: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:23 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:24.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 259401 12.16 MB/sec execute 9 sec latency 23.034 ms 2024-03-18T16:03:25.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 261570 11.75 MB/sec execute 10 sec latency 59.996 ms 2024-03-18T16:03:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:25 smithi191 ceph-mon[36784]: pgmap v103: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:25 smithi156 ceph-mon[35919]: pgmap v103: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:25 smithi150 ceph-mon[31946]: pgmap v103: 97 pgs: 97 active+clean; 138 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 226 op/s 2024-03-18T16:03:26.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 263455 11.76 MB/sec execute 11 sec latency 17.823 ms 2024-03-18T16:03:27.318 INFO:tasks.workunit.client.0.smithi150.stdout: 1 266268 12.28 MB/sec execute 12 sec latency 12.693 ms 2024-03-18T16:03:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:27 smithi191 ceph-mon[36784]: pgmap v104: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 336 op/s 2024-03-18T16:03:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:27 smithi156 ceph-mon[35919]: pgmap v104: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 336 op/s 2024-03-18T16:03:28.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:27 smithi150 ceph-mon[31946]: pgmap v104: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 336 op/s 2024-03-18T16:03:28.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 268496 11.91 MB/sec execute 13 sec latency 14.273 ms 2024-03-18T16:03:29.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:28 smithi156 ceph-mon[35919]: pgmap v105: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 221 op/s 2024-03-18T16:03:29.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:28 smithi150 ceph-mon[31946]: pgmap v105: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 221 op/s 2024-03-18T16:03:29.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:28 smithi191 ceph-mon[36784]: pgmap v105: 97 pgs: 97 active+clean; 144 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 221 op/s 2024-03-18T16:03:29.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 270357 11.94 MB/sec execute 14 sec latency 19.394 ms 2024-03-18T16:03:30.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 272600 11.96 MB/sec execute 15 sec latency 70.595 ms 2024-03-18T16:03:31.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 274561 11.94 MB/sec execute 16 sec latency 16.701 ms 2024-03-18T16:03:31.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:31 smithi150 ceph-mon[31946]: pgmap v106: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 325 op/s 2024-03-18T16:03:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:31 smithi191 ceph-mon[36784]: pgmap v106: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 325 op/s 2024-03-18T16:03:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:31 smithi156 ceph-mon[35919]: pgmap v106: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 325 op/s 2024-03-18T16:03:32.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 277136 12.06 MB/sec execute 17 sec latency 10.015 ms 2024-03-18T16:03:33.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 279608 12.09 MB/sec execute 18 sec latency 24.220 ms 2024-03-18T16:03:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:33 smithi191 ceph-mon[36784]: pgmap v107: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:33.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:33 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:33 smithi156 ceph-mon[35919]: pgmap v107: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:33 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:33 smithi150 ceph-mon[31946]: pgmap v107: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:33 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:34.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 281398 11.91 MB/sec execute 19 sec latency 13.477 ms 2024-03-18T16:03:35.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 283857 12.09 MB/sec execute 20 sec latency 81.163 ms 2024-03-18T16:03:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:35 smithi191 ceph-mon[36784]: pgmap v108: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:35 smithi156 ceph-mon[35919]: pgmap v108: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:35 smithi150 ceph-mon[31946]: pgmap v108: 97 pgs: 97 active+clean; 149 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-03-18T16:03:36.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 285560 11.84 MB/sec execute 21 sec latency 68.930 ms 2024-03-18T16:03:36.986 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 -- 172.21.15.150:0/2844916521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c1067e0 msgr2=0x7f591c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 --2- 172.21.15.150:0/2844916521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c1067e0 0x7f591c106bc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5918009900 tx=0x7f591802bc30 comp rx=0 tx=0).stop 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 -- 172.21.15.150:0/2844916521 shutdown_connections 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 --2- 172.21.15.150:0/2844916521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c101380 0x7f591c10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 --2- 172.21.15.150:0/2844916521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 0x7f591c100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 --2- 172.21.15.150:0/2844916521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c1067e0 0x7f591c106bc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 -- 172.21.15.150:0/2844916521 >> 172.21.15.150:0/2844916521 conn(0x7f591c0774b0 msgr2=0x7f591c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 -- 172.21.15.150:0/2844916521 shutdown_connections 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.983+0000 7f5923a9a640 1 -- 172.21.15.150:0/2844916521 wait complete. 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 Processor -- start 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 -- start start 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 0x7f591c196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c1067e0 0x7f591c19b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f591c110ae0 con 0x7f591c101380 2024-03-18T16:03:36.987 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f591c110960 con 0x7f591c1067e0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5923a9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f591c110c60 con 0x7f591c1009c0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39028/0 (socket says 172.21.15.150:39028) 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 -- 172.21.15.150:0/1505321382 learned_addr learned my addr 172.21.15.150:0/1505321382 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921836640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 0x7f591c196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5922037640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c1067e0 0x7f591c19b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 -- 172.21.15.150:0/1505321382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 msgr2=0x7f591c196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 0x7f591c196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 -- 172.21.15.150:0/1505321382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c1067e0 msgr2=0x7f591c19b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c1067e0 0x7f591c19b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.984+0000 7f5921035640 1 -- 172.21.15.150:0/1505321382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c19be10 con 0x7f591c101380 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5921035640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f590c00e9f0 tx=0x7f590c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5912ffd640 1 -- 172.21.15.150:0/1505321382 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c00cd80 con 0x7f591c101380 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c19c100 con 0x7f591c101380 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5912ffd640 1 -- 172.21.15.150:0/1505321382 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590c005500 con 0x7f591c101380 2024-03-18T16:03:36.988 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5912ffd640 1 -- 172.21.15.150:0/1505321382 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c010690 con 0x7f591c101380 2024-03-18T16:03:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.985+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c1a3990 con 0x7f591c101380 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.987+0000 7f5912ffd640 1 -- 172.21.15.150:0/1505321382 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f590c0046c0 con 0x7f591c101380 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.987+0000 7f5926763740 1 Processor -- start 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 -- start start 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5912ffd640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 0x7f59080a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff64510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a98fec8a40 con 0x55a98ff6d820 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a98fe71920 con 0x55a98ff64130 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5926763740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a98fec4ae0 con 0x55a98ff64a50 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5912ffd640 1 -- 172.21.15.150:0/1505321382 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f590c0cf200 con 0x7f591c101380 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5922037640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 0x7f59080a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921035640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff64510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921035640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:34734/0 (socket says 172.21.15.150:34734) 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921035640 1 -- 172.21.15.150:0/2816803983 learned_addr learned my addr 172.21.15.150:0/2816803983 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5922037640 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 -- 172.21.15.150:0/2816803983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 msgr2=0x55a98ff6d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 -- 172.21.15.150:0/2816803983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 msgr2=0x55a98ff6fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921836640 1 -- 172.21.15.150:0/2816803983 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a98fe89460 con 0x55a98ff64130 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5921035640 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.988+0000 7f5922037640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 0x7f59080a4ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5914005fd0 tx=0x7f5914009f00 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5921836640 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff64510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5918000c00 tx=0x7f5918005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82e5dd8f770fc6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5907fff640 1 -- 172.21.15.150:0/2816803983 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918042070 con 0x55a98ff64130 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5907fff640 1 -- 172.21.15.150:0/2816803983 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591802c8b0 con 0x55a98ff64130 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5907fff640 1 -- 172.21.15.150:0/2816803983 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918004b60 con 0x55a98ff64130 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5926763740 1 -- 172.21.15.150:0/2816803983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 msgr2=0x55a98ff64510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.989+0000 7f5926763740 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff64510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5918000c00 tx=0x7f5918005be0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 -- 172.21.15.150:0/2816803983 shutdown_connections 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64a50 0x55a98ff6d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 --2- 172.21.15.150:0/2816803983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff64510 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 -- 172.21.15.150:0/2816803983 >> 172.21.15.150:0/2816803983 conn(0x55a98ff5ea50 msgr2=0x55a98ff5ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 -- 172.21.15.150:0/2816803983 shutdown_connections 2024-03-18T16:03:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 -- 172.21.15.150:0/2816803983 wait complete. 2024-03-18T16:03:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 -- start start 2024-03-18T16:03:36.994 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.990+0000 7f5926763740 1 Processor -- start 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff6a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64a50 0x55a98ff6ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a98ff703f0 con 0x55a98ff6d820 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a98fe84d70 con 0x55a98ff64a50 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55a98ff70570 con 0x55a98ff64130 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 10 client.0 _ll_register_callbacks cb 0x55a98fbdca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5921836640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff6a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5921035640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64a50 0x55a98ff6ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39050/0 (socket says 172.21.15.150:39050) 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 -- 172.21.15.150:0/1195060452 learned_addr learned my addr 172.21.15.150:0/1195060452 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 -- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 msgr2=0x55a98ff6a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff6a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 -- 172.21.15.150:0/1195060452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64a50 msgr2=0x55a98ff6ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64a50 0x55a98ff6ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 -- 172.21.15.150:0/1195060452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a98fbdd210 con 0x55a98ff6d820 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5922037640 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f591401e570 tx=0x7f591401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148380c9b8eaef16 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f59057fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914024070 con 0x55a98ff6d820 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591802c8b0 con 0x55a98ff6d820 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f5914004b20 con 0x55a98ff6d820 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5914004d20 con 0x55a98ff6d820 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 10 client.15702 fetch_fsmap learned FSMap version 28 2024-03-18T16:03:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.991+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a98fca9620 con 0x55a98ff6d820 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591802c8b0 con 0x55a98ff6d820 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8342+0+0 (secure 0 0 0) 0x7f59140050b0 con 0x55a98ff6d820 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 10 client.15702 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 20 client.15702 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ff7fba0 0x55a98ff81fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 4 client.15702 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a98fed5f80 con 0x55a98ff7fba0 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5921836640 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ff7fba0 0x55a98ff81fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f59057fa640 10 client.15702 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.992+0000 7f5921836640 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ff7fba0 0x55a98ff81fb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb73dbb6f8f2ce2 server_cookie=c12a69480a259d71 in_seq=0 out_seq=0 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f59057fa640 1 -- 172.21.15.150:0/1195060452 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a98fed5f80 con 0x55a98ff7fba0 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f59057fa640 10 client.15702 handle_command_reply: tid=0 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 client.15702 shutdown 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 msgr2=0x55a98ff6b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f591401e570 tx=0x7f591401d000 comp rx=0 tx=0).stop 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 shutdown_connections 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ff7fba0 0x55a98ff81fb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff6d820 0x55a98ff6b320 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64a50 0x55a98ff6ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f59057fa640 0 client.15702 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 --2- 172.21.15.150:0/1195060452 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff6a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 >> 172.21.15.150:0/1195060452 conn(0x55a98ff5ea50 msgr2=0x55a98ff607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 shutdown_connections 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.993+0000 7f5926763740 1 -- 172.21.15.150:0/1195060452 wait complete. 2024-03-18T16:03:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:36.994+0000 7f5926763740 20 client.15702 trim_cache size 0 max 16384 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 Processor -- start 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 -- start start 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff6cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 0x55a98ff79190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1ad890 con 0x55a98ff64a50 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1ab160 con 0x55a98ff64130 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5926763740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1a8fb0 con 0x55a98ff6d820 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.011 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39058/0 (socket says 172.21.15.150:39058) 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 -- 172.21.15.150:0/2357884294 learned_addr learned my addr 172.21.15.150:0/2357884294 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921836640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff6cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5922037640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 0x55a98ff79190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 -- 172.21.15.150:0/2357884294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 msgr2=0x55a98ff79190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 0x55a98ff79190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 -- 172.21.15.150:0/2357884294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 msgr2=0x55a98ff6cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff6cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5921035640 1 -- 172.21.15.150:0/2357884294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a98ff5a310 con 0x55a98ff64a50 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.008+0000 7f5922037640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 0x55a98ff79190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5921035640 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f590c075fc0 tx=0x7f590c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc4dd91ae4c3ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f59077fe640 1 -- 172.21.15.150:0/2357884294 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c09b8b0 con 0x55a98ff64a50 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f59077fe640 1 -- 172.21.15.150:0/2357884294 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590c09fa00 con 0x55a98ff64a50 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f59077fe640 1 -- 172.21.15.150:0/2357884294 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c0860b0 con 0x55a98ff64a50 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 -- 172.21.15.150:0/2357884294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 msgr2=0x55a98ff59a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:37.012 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f590c075fc0 tx=0x7f590c0056b0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 -- 172.21.15.150:0/2357884294 shutdown_connections 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff6d820 0x55a98ff79190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff59a80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 --2- 172.21.15.150:0/2357884294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff64130 0x55a98ff6cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 -- 172.21.15.150:0/2357884294 >> 172.21.15.150:0/2357884294 conn(0x55a98ff5ea50 msgr2=0x55a98ff5fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 -- 172.21.15.150:0/2357884294 shutdown_connections 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.009+0000 7f5926763740 1 -- 172.21.15.150:0/2357884294 wait complete. 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 -- start start 2024-03-18T16:03:37.013 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 Processor -- start 2024-03-18T16:03:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff6d820 0x55a98ff72cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1a9620 con 0x55a98ff64a50 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1aaa90 con 0x55a98ff6d820 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1a9cf0 con 0x55a98ff64130 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5926763740 10 client.0 _ll_register_callbacks cb 0x7f5914009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39064/0 (socket says 172.21.15.150:39064) 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 -- 172.21.15.150:0/2336470684 learned_addr learned my addr 172.21.15.150:0/2336470684 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921836640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5922037640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff6d820 0x55a98ff72cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 -- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 msgr2=0x55a98ff759b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff759b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 -- 172.21.15.150:0/2336470684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff6d820 msgr2=0x55a98ff72cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff6d820 0x55a98ff72cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.010+0000 7f5921035640 1 -- 172.21.15.150:0/2336470684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a98ff73240 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5921035640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f590c0cca50 tx=0x7f590c0ccc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd3bcc5ecd148e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c0b8020 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f590c0760e0 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590c09fa00 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f590c0862b0 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 10 client.15708 fetch_fsmap learned FSMap version 28 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a98ffe6bd0 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590c09fa00 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) ==== 8342+0+0 (secure 0 0 0) 0x7f590c09bb10 con 0x55a98ff64a50 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 10 client.15708 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 20 client.15708 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ffe71b0 0x55a98ffe95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 4 client.15708 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:03:37.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a98fed02d0 con 0x55a98ffe71b0 2024-03-18T16:03:37.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.011+0000 7f5921836640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ffe71b0 0x55a98ffe95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:37.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.012+0000 7f59057fa640 10 client.15708 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:37.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.012+0000 7f5921836640 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ffe71b0 0x55a98ffe95d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da33c9a4932ae75c server_cookie=b3b2fcb58910b27b in_seq=0 out_seq=0 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f59057fa640 1 -- 172.21.15.150:0/2336470684 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a98fed02d0 con 0x55a98ffe71b0 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f59057fa640 10 client.15708 handle_command_reply: tid=0 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 client.15708 shutdown 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 msgr2=0x55a98ff72790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f590c0cca50 tx=0x7f590c0ccc90 comp rx=0 tx=0).stop 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 shutdown_connections 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55a98ffe71b0 0x55a98ffe95d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a98ff6d820 0x55a98ff72cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a98ff64a50 0x55a98ff72790 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f59057fa640 0 client.15708 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 --2- 172.21.15.150:0/2336470684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a98ff64130 0x55a98ff759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 >> 172.21.15.150:0/2336470684 conn(0x55a98ff5ea50 msgr2=0x55a98ff6b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 shutdown_connections 2024-03-18T16:03:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 1 -- 172.21.15.150:0/2336470684 wait complete. 2024-03-18T16:03:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.020+0000 7f5926763740 20 client.15708 trim_cache size 0 max 16384 2024-03-18T16:03:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:03:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:03:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:03:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 msgr2=0x7f59080a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 0x7f59080a4ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5914005fd0 tx=0x7f5914009f00 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 msgr2=0x7f591c197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f590c00e9f0 tx=0x7f590c00edf0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 shutdown_connections 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f59080a2a10 0x7f59080a4ed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f591c1067e0 0x7f591c19b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f591c101380 0x7f591c197360 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 --2- 172.21.15.150:0/1505321382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f591c1009c0 0x7f591c196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 >> 172.21.15.150:0/1505321382 conn(0x7f591c0774b0 msgr2=0x7f591c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 shutdown_connections 2024-03-18T16:03:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:37.021+0000 7f5923a9a640 1 -- 172.21.15.150:0/1505321382 wait complete. 2024-03-18T16:03:37.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 287920 11.93 MB/sec execute 22 sec latency 12.916 ms 2024-03-18T16:03:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:37 smithi191 ceph-mon[36784]: pgmap v109: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 318 op/s 2024-03-18T16:03:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:37 smithi156 ceph-mon[35919]: pgmap v109: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 318 op/s 2024-03-18T16:03:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:37 smithi150 ceph-mon[31946]: pgmap v109: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 318 op/s 2024-03-18T16:03:38.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 290528 11.99 MB/sec execute 23 sec latency 24.713 ms 2024-03-18T16:03:39.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 292433 11.94 MB/sec execute 24 sec latency 15.810 ms 2024-03-18T16:03:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:39 smithi191 ceph-mon[36784]: pgmap v110: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:03:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:39 smithi156 ceph-mon[35919]: pgmap v110: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:03:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:39 smithi150 ceph-mon[31946]: pgmap v110: 97 pgs: 97 active+clean; 156 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:03:40.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 294783 12.03 MB/sec execute 25 sec latency 64.886 ms 2024-03-18T16:03:41.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 297150 11.99 MB/sec execute 26 sec latency 18.793 ms 2024-03-18T16:03:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:41 smithi191 ceph-mon[36784]: pgmap v111: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 311 op/s 2024-03-18T16:03:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:41 smithi156 ceph-mon[35919]: pgmap v111: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 311 op/s 2024-03-18T16:03:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:41 smithi150 ceph-mon[31946]: pgmap v111: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 311 op/s 2024-03-18T16:03:42.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 299032 11.92 MB/sec execute 27 sec latency 19.274 ms 2024-03-18T16:03:43.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 301823 12.09 MB/sec execute 28 sec latency 11.072 ms 2024-03-18T16:03:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:43 smithi191 ceph-mon[36784]: pgmap v112: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:43 smithi156 ceph-mon[35919]: pgmap v112: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:43 smithi150 ceph-mon[31946]: pgmap v112: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:44.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 304020 11.95 MB/sec execute 29 sec latency 16.646 ms 2024-03-18T16:03:45.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 305685 11.93 MB/sec execute 30 sec latency 106.126 ms 2024-03-18T16:03:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:45 smithi191 ceph-mon[36784]: pgmap v113: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:45 smithi156 ceph-mon[35919]: pgmap v113: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:45 smithi150 ceph-mon[31946]: pgmap v113: 97 pgs: 97 active+clean; 166 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-03-18T16:03:46.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 308271 11.98 MB/sec execute 31 sec latency 27.912 ms 2024-03-18T16:03:47.319 INFO:tasks.workunit.client.0.smithi150.stdout: 1 310066 11.94 MB/sec execute 32 sec latency 42.225 ms 2024-03-18T16:03:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:47 smithi191 ceph-mon[36784]: pgmap v114: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 315 op/s 2024-03-18T16:03:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:47 smithi156 ceph-mon[35919]: pgmap v114: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 315 op/s 2024-03-18T16:03:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:47 smithi150 ceph-mon[31946]: pgmap v114: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 315 op/s 2024-03-18T16:03:48.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 312558 12.01 MB/sec execute 33 sec latency 30.155 ms 2024-03-18T16:03:48.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:48 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:49.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:48 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:49.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:48 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:03:49.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 314906 11.98 MB/sec execute 34 sec latency 19.108 ms 2024-03-18T16:03:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:49 smithi191 ceph-mon[36784]: pgmap v115: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-18T16:03:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:49 smithi156 ceph-mon[35919]: pgmap v115: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-18T16:03:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:49 smithi150 ceph-mon[31946]: pgmap v115: 97 pgs: 97 active+clean; 172 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 -- 172.21.15.150:0/1733116571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 msgr2=0x7fc7ac100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/1733116571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac100db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009900 tx=0x7fc79c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 -- 172.21.15.150:0/1733116571 shutdown_connections 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/1733116571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac1012f0 0x7fc7ac10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/1733116571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac100db0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/1733116571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac105680 0x7fc7ac105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 -- 172.21.15.150:0/1733116571 >> 172.21.15.150:0/1733116571 conn(0x7fc7ac077530 msgr2=0x7fc7ac077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 -- 172.21.15.150:0/1733116571 shutdown_connections 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.108+0000 7fc7b45c4640 1 -- 172.21.15.150:0/1733116571 wait complete. 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 Processor -- start 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 -- start start 2024-03-18T16:03:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac1012f0 0x7fc7ac1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac105680 0x7fc7ac19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac110ba0 con 0x7fc7ac100930 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac110a20 con 0x7fc7ac105680 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b45c4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac110d20 con 0x7fc7ac1012f0 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34366/0 (socket says 172.21.15.150:34366) 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 -- 172.21.15.150:0/3646269072 learned_addr learned my addr 172.21.15.150:0/3646269072 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2b61640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac105680 0x7fc7ac19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac1012f0 0x7fc7ac1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 -- 172.21.15.150:0/3646269072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac1012f0 msgr2=0x7fc7ac1974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac1012f0 0x7fc7ac1974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 -- 172.21.15.150:0/3646269072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac105680 msgr2=0x7fc7ac19b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac105680 0x7fc7ac19b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.109+0000 7fc7b2360640 1 -- 172.21.15.150:0/3646269072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac19bf00 con 0x7fc7ac100930 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7b2360640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9f0 tx=0x7fc7a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7a37fe640 1 -- 172.21.15.150:0/3646269072 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800cd80 con 0x7fc7ac100930 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7a37fe640 1 -- 172.21.15.150:0/3646269072 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a8005500 con 0x7fc7ac100930 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac19c1f0 con 0x7fc7ac100930 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7a37fe640 1 -- 172.21.15.150:0/3646269072 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8010690 con 0x7fc7ac100930 2024-03-18T16:03:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.110+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ac1a3a80 con 0x7fc7ac100930 2024-03-18T16:03:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.112+0000 7fc7a37fe640 1 -- 172.21.15.150:0/3646269072 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fc7a80046c0 con 0x7fc7ac100930 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 Processor -- start 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 -- start start 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a8a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 0x563cb1a95c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563cb19eea40 con 0x563cb1a93820 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563cb1997920 con 0x563cb1a8a130 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563cb19eaae0 con 0x563cb1a8aa50 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7a37fe640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 0x7fc7980a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b1b5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 0x563cb1a95c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b2b61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a8a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b1b5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 0x563cb1a95c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34368/0 (socket says 172.21.15.150:34368) 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.113+0000 7fc7b1b5f640 1 -- 172.21.15.150:0/1464137267 learned_addr learned my addr 172.21.15.150:0/1464137267 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2b61640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 0x7fc7980a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7a37fe640 1 -- 172.21.15.150:0/3646269072 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc7a80cf200 con 0x7fc7ac100930 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 -- 172.21.15.150:0/1464137267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 msgr2=0x563cb1a8a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a8a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 -- 172.21.15.150:0/1464137267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 msgr2=0x563cb1a95c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 0x563cb1a95c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 -- 172.21.15.150:0/1464137267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cb19af460 con 0x563cb1a8aa50 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2b61640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 0x7fc7980a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4005c40 tx=0x7fc7a4009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a932e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cc9a0 tx=0x7fc7a806e650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f71936d9545ef6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7a0bfe640 1 -- 172.21.15.150:0/1464137267 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a809b210 con 0x563cb1a8aa50 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7a0bfe640 1 -- 172.21.15.150:0/1464137267 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a809bfb0 con 0x563cb1a8aa50 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.114+0000 7fc7a0bfe640 1 -- 172.21.15.150:0/1464137267 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80860f0 con 0x563cb1a8aa50 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1464137267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 msgr2=0x563cb1a932e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a932e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cc9a0 tx=0x7fc7a806e650 comp rx=0 tx=0).stop 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1464137267 shutdown_connections 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a93820 0x563cb1a95c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a932e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1464137267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a8a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1464137267 >> 172.21.15.150:0/1464137267 conn(0x563cb1a84a50 msgr2=0x563cb1a84e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1464137267 shutdown_connections 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1464137267 wait complete. 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 -- start start 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.115+0000 7fc7b71a9740 1 Processor -- start 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a90810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a91290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563cb1a963f0 con 0x563cb1a8aa50 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563cb19aad70 con 0x563cb1a8a130 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563cb1a96570 con 0x563cb1a93820 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 10 client.0 _ll_register_callbacks cb 0x563cb1702a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a90810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b2b61640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a91290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34374/0 (socket says 172.21.15.150:34374) 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 -- 172.21.15.150:0/2300715166 learned_addr learned my addr 172.21.15.150:0/2300715166 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a90810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:55380/0 (socket says 172.21.15.150:55380) 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 -- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 msgr2=0x563cb1a91290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a91290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 -- 172.21.15.150:0/2300715166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 msgr2=0x563cb1a90810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a90810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 -- 172.21.15.150:0/2300715166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cb1703210 con 0x563cb1a8aa50 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009230 tx=0x7fc79c02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ff4416f3a6f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c02ccc0 con 0x563cb1a8aa50 2024-03-18T16:03:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc79c0046c0 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7a809bfb0 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc79c0048c0 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 10 client.15720 fetch_fsmap learned FSMap version 29 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.116+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cb17cf620 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a809bfb0 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc79c046070 con 0x563cb1a8aa50 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 10 client.15720 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 20 client.15720 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1aa38a0 0x563cb1aa5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 4 client.15720 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cb19fbf80 con 0x563cb1aa38a0 2024-03-18T16:03:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b2360640 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1aa38a0 0x563cb1aa5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc795ffb640 10 client.15720 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.117+0000 7fc7b2360640 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1aa38a0 0x563cb1aa5d10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27c183bf845bbfef server_cookie=d4e775517bf408ef in_seq=0 out_seq=0 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc795ffb640 1 -- 172.21.15.150:0/2300715166 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563cb19fbf80 con 0x563cb1aa38a0 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc795ffb640 10 client.15720 handle_command_reply: tid=0 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 client.15720 shutdown 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 msgr2=0x563cb1a90d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009230 tx=0x7fc79c02c800 comp rx=0 tx=0).stop 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 shutdown_connections 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1aa38a0 0x563cb1aa5d10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a91290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8aa50 0x563cb1a90d50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/2300715166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8a130 0x563cb1a90810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc795ffb640 0 client.15720 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 >> 172.21.15.150:0/2300715166 conn(0x563cb1a84a50 msgr2=0x563cb1a867a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 shutdown_connections 2024-03-18T16:03:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 1 -- 172.21.15.150:0/2300715166 wait complete. 2024-03-18T16:03:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.118+0000 7fc7b71a9740 20 client.15720 trim_cache size 0 max 16384 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 Processor -- start 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 -- start start 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a9f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a93820 0x563cb1aab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1aea20 con 0x563cb1a8a130 2024-03-18T16:03:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1ac2f0 con 0x563cb1a93820 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1aa140 con 0x563cb1a8aa50 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34376/0 (socket says 172.21.15.150:34376) 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 -- 172.21.15.150:0/1612325197 learned_addr learned my addr 172.21.15.150:0/1612325197 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a9f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2b61640 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a93820 0x563cb1aab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 -- 172.21.15.150:0/1612325197 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 msgr2=0x563cb1a9f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a9f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 -- 172.21.15.150:0/1612325197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a93820 msgr2=0x563cb1aab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a93820 0x563cb1aab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.133+0000 7fc7b2360640 1 -- 172.21.15.150:0/1612325197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cb1a9f6d0 con 0x563cb1a8a130 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b2360640 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8097d10 tx=0x7fc7a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e7caa69db4c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc797fff640 1 -- 172.21.15.150:0/1612325197 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80b3070 con 0x563cb1a8a130 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc797fff640 1 -- 172.21.15.150:0/1612325197 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a809fa00 con 0x563cb1a8a130 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc797fff640 1 -- 172.21.15.150:0/1612325197 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80860f0 con 0x563cb1a8a130 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1612325197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 msgr2=0x563cb1a92f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8097d10 tx=0x7fc7a8004a40 comp rx=0 tx=0).stop 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1612325197 shutdown_connections 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a93820 0x563cb1aab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a8aa50 0x563cb1a9f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/1612325197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a92f40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1612325197 >> 172.21.15.150:0/1612325197 conn(0x563cb1a84a50 msgr2=0x563cb1a907d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1612325197 shutdown_connections 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.134+0000 7fc7b71a9740 1 -- 172.21.15.150:0/1612325197 wait complete. 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 -- start start 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 Processor -- start 2024-03-18T16:03:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 0x563cb1a977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a97d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1aa7b0 con 0x563cb1a8a130 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1abc20 con 0x563cb1a8aa50 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac1aae80 con 0x563cb1a93820 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b71a9740 10 client.0 _ll_register_callbacks cb 0x7fc79c004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b1b5f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 0x563cb1a977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34390/0 (socket says 172.21.15.150:34390) 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 -- 172.21.15.150:0/872527333 learned_addr learned my addr 172.21.15.150:0/872527333 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2b61640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a97d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 -- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 msgr2=0x563cb1a97d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a97d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 -- 172.21.15.150:0/872527333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 msgr2=0x563cb1a977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 0x563cb1a977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b2360640 1 -- 172.21.15.150:0/872527333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cb1b0d260 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.135+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 0x563cb1a977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b2360640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9c0 tx=0x7fc7a809b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7819f976022b418c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80b3050 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563cb19f62d0 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc7a80860f0 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7a809bfb0 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 10 client.15726 fetch_fsmap learned FSMap version 29 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cb1b0e530 con 0x563cb1a8a130 2024-03-18T16:03:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563cb19f62d0 con 0x563cb1a8a130 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc7a80a3bc0 con 0x563cb1a8a130 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 10 client.15726 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 20 client.15726 populate_metadata read hostname 'smithi150' 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1b0e760 0x563cb1b10b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 4 client.15726 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cb1974770 con 0x563cb1b0e760 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.136+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1b0e760 0x563cb1b10b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.137+0000 7fc7b1b5f640 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1b0e760 0x563cb1b10b50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60735940680537ad server_cookie=6215941be5088f4d in_seq=0 out_seq=0 2024-03-18T16:03:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.137+0000 7fc795ffb640 10 client.15726 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc795ffb640 1 -- 172.21.15.150:0/872527333 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563cb1974770 con 0x563cb1b0e760 2024-03-18T16:03:50.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc795ffb640 10 client.15726 handle_command_reply: tid=0 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 client.15726 shutdown 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 msgr2=0x563cb1a97290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9c0 tx=0x7fc7a809b700 comp rx=0 tx=0).stop 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 shutdown_connections 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563cb1b0e760 0x563cb1b10b50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563cb1a93820 0x563cb1a97d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cb1a8aa50 0x563cb1a977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 --2- 172.21.15.150:0/872527333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563cb1a8a130 0x563cb1a97290 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc795ffb640 0 client.15726 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.145+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 >> 172.21.15.150:0/872527333 conn(0x563cb1a84a50 msgr2=0x563cb1a83bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 shutdown_connections 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b71a9740 1 -- 172.21.15.150:0/872527333 wait complete. 2024-03-18T16:03:50.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b71a9740 20 client.15726 trim_cache size 0 max 16384 2024-03-18T16:03:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:03:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:03:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:03:50.149 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 msgr2=0x7fc7980a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 0x7fc7980a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4005c40 tx=0x7fc7a4009500 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 msgr2=0x7fc7ac196f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800e9f0 tx=0x7fc7a800edf0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 shutdown_connections 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc7980a2a10 0x7fc7980a4ed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7ac105680 0x7fc7ac19b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7ac1012f0 0x7fc7ac1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 --2- 172.21.15.150:0/3646269072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7ac100930 0x7fc7ac196f60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 >> 172.21.15.150:0/3646269072 conn(0x7fc7ac077530 msgr2=0x7fc7ac0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 shutdown_connections 2024-03-18T16:03:50.150 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:03:50.146+0000 7fc7b45c4640 1 -- 172.21.15.150:0/3646269072 wait complete. 2024-03-18T16:03:50.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 316489 11.88 MB/sec execute 35 sec latency 118.363 ms 2024-03-18T16:03:51.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 319059 11.94 MB/sec execute 36 sec latency 16.708 ms 2024-03-18T16:03:51.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:51 smithi191 ceph-mon[36784]: pgmap v116: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-18T16:03:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:51 smithi156 ceph-mon[35919]: pgmap v116: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-18T16:03:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:51 smithi150 ceph-mon[31946]: pgmap v116: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-18T16:03:52.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 320895 11.88 MB/sec execute 37 sec latency 11.414 ms 2024-03-18T16:03:53.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 323347 11.94 MB/sec execute 38 sec latency 9.189 ms 2024-03-18T16:03:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:53 smithi191 ceph-mon[36784]: pgmap v117: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:53 smithi156 ceph-mon[35919]: pgmap v117: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:53 smithi150 ceph-mon[31946]: pgmap v117: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:54.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 325807 11.95 MB/sec execute 39 sec latency 17.881 ms 2024-03-18T16:03:55.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 327317 11.84 MB/sec execute 40 sec latency 90.717 ms 2024-03-18T16:03:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:55 smithi191 ceph-mon[36784]: pgmap v118: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:55 smithi156 ceph-mon[35919]: pgmap v118: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:55 smithi150 ceph-mon[31946]: pgmap v118: 97 pgs: 97 active+clean; 178 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-03-18T16:03:56.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 329972 11.96 MB/sec execute 41 sec latency 23.304 ms 2024-03-18T16:03:57.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 330864 11.73 MB/sec execute 42 sec latency 294.176 ms 2024-03-18T16:03:57.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:57 smithi191 ceph-mon[36784]: pgmap v119: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 329 op/s 2024-03-18T16:03:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:57 smithi156 ceph-mon[35919]: pgmap v119: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 329 op/s 2024-03-18T16:03:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:57 smithi150 ceph-mon[31946]: pgmap v119: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 329 op/s 2024-03-18T16:03:58.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 333588 11.85 MB/sec execute 43 sec latency 12.927 ms 2024-03-18T16:03:59.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 335812 11.77 MB/sec execute 44 sec latency 13.938 ms 2024-03-18T16:03:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:03:59 smithi191 ceph-mon[36784]: pgmap v120: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 221 op/s 2024-03-18T16:04:00.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:03:59 smithi156 ceph-mon[35919]: pgmap v120: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 221 op/s 2024-03-18T16:04:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:03:59 smithi150 ceph-mon[31946]: pgmap v120: 97 pgs: 97 active+clean; 187 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 221 op/s 2024-03-18T16:04:00.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 337430 11.78 MB/sec execute 45 sec latency 79.742 ms 2024-03-18T16:04:01.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 338786 11.64 MB/sec execute 46 sec latency 40.291 ms 2024-03-18T16:04:01.894 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:01 smithi150 ceph-mon[31946]: pgmap v121: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-03-18T16:04:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:01 smithi191 ceph-mon[36784]: pgmap v121: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-03-18T16:04:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:01 smithi156 ceph-mon[35919]: pgmap v121: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 334 op/s 2024-03-18T16:04:02.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 341053 11.69 MB/sec execute 47 sec latency 9.078 ms 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 -- 172.21.15.150:0/346695418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c0100930 msgr2=0x7f58c0100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/346695418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c0100930 0x7f58c0100db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009890 tx=0x7f58b002bb70 comp rx=0 tx=0).stop 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 -- 172.21.15.150:0/346695418 shutdown_connections 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/346695418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c01012f0 0x7f58c010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/346695418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c0100930 0x7f58c0100db0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/346695418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0105680 0x7f58c0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 -- 172.21.15.150:0/346695418 >> 172.21.15.150:0/346695418 conn(0x7f58c0077530 msgr2=0x7f58c0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 -- 172.21.15.150:0/346695418 shutdown_connections 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.237+0000 7f58c7c8c640 1 -- 172.21.15.150:0/346695418 wait complete. 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 Processor -- start 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 -- start start 2024-03-18T16:04:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0100930 0x7f58c0197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c01012f0 0x7f58c0197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0110ba0 con 0x7f58c0105680 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0110a20 con 0x7f58c01012f0 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c7c8c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58c0110d20 con 0x7f58c0100930 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c6229640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c6229640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45952/0 (socket says 172.21.15.150:45952) 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0100930 0x7f58c0197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c6229640 1 -- 172.21.15.150:0/2622042807 learned_addr learned my addr 172.21.15.150:0/2622042807 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.238+0000 7f58c5227640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c01012f0 0x7f58c0197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 -- 172.21.15.150:0/2622042807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0100930 msgr2=0x7f58c0197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0100930 0x7f58c0197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 -- 172.21.15.150:0/2622042807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c01012f0 msgr2=0x7f58c0197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c01012f0 0x7f58c0197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 -- 172.21.15.150:0/2622042807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c019bfc0 con 0x7f58c0105680 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c6229640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f58b800eee0 tx=0x7f58b800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58b6ffd640 1 -- 172.21.15.150:0/2622042807 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8004b20 con 0x7f58c0105680 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58b6ffd640 1 -- 172.21.15.150:0/2622042807 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b8003e50 con 0x7f58c0105680 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58b6ffd640 1 -- 172.21.15.150:0/2622042807 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8005280 con 0x7f58c0105680 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c019c2b0 con 0x7f58c0105680 2024-03-18T16:04:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.239+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c01a3b40 con 0x7f58c0105680 2024-03-18T16:04:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.241+0000 7f58b6ffd640 1 -- 172.21.15.150:0/2622042807 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f58b801d020 con 0x7f58c0105680 2024-03-18T16:04:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.242+0000 7f58ca87f740 1 Processor -- start 2024-03-18T16:04:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.242+0000 7f58b6ffd640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 0x7f58ac0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.242+0000 7f58ca87f740 1 -- start start 2024-03-18T16:04:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5227640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 0x7f58ac0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58b6ffd640 1 -- 172.21.15.150:0/2622042807 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f58b80cef40 con 0x7f58c0105680 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe3a530 0x55c78fe454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c78fd9ea40 con 0x55c78fe39c10 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c78fd47920 con 0x55c78fe45a30 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58ca87f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c78fd9aae0 con 0x55c78fe3a530 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45966/0 (socket says 172.21.15.150:45966) 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:55468/0 (socket says 172.21.15.150:55468) 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5a28640 1 -- 172.21.15.150:0/665825859 learned_addr learned my addr 172.21.15.150:0/665825859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5227640 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe3a530 0x55c78fe454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 -- 172.21.15.150:0/665825859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe3a530 msgr2=0x55c78fe454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe3a530 0x55c78fe454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 -- 172.21.15.150:0/665825859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 msgr2=0x55c78fe39ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 -- 172.21.15.150:0/665825859 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78fd5f460 con 0x55c78fe45a30 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5227640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 0x7f58ac0a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000c00 tx=0x7f58b00323c0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c5a28640 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58c6229640 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58b80cc6e0 tx=0x7f58b8004020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d5dde1b65d82d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.243+0000 7f58abfff640 1 -- 172.21.15.150:0/665825859 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b801a040 con 0x55c78fe45a30 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58abfff640 1 -- 172.21.15.150:0/665825859 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b809b550 con 0x55c78fe45a30 2024-03-18T16:04:03.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58abfff640 1 -- 172.21.15.150:0/665825859 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8004cc0 con 0x55c78fe45a30 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 -- 172.21.15.150:0/665825859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 msgr2=0x55c78fe47e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f58b80cc6e0 tx=0x7f58b8004020 comp rx=0 tx=0).stop 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 -- 172.21.15.150:0/665825859 shutdown_connections 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe47e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe3a530 0x55c78fe454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 --2- 172.21.15.150:0/665825859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe39c10 0x55c78fe39ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 -- 172.21.15.150:0/665825859 >> 172.21.15.150:0/665825859 conn(0x55c78fe34a50 msgr2=0x55c78fe34e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 -- 172.21.15.150:0/665825859 shutdown_connections 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.244+0000 7f58ca87f740 1 -- 172.21.15.150:0/665825859 wait complete. 2024-03-18T16:04:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 -- start start 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 Processor -- start 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe41100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78fe537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c78fe3b520 con 0x55c78fe3a530 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c78fd5ad70 con 0x55c78fe39c10 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55c78fe3b6a0 con 0x55c78fe45a30 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58ca87f740 10 client.0 _ll_register_callbacks cb 0x55c78fab2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45980/0 (socket says 172.21.15.150:45980) 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58c5227640 1 -- 172.21.15.150:0/1756871982 learned_addr learned my addr 172.21.15.150:0/1756871982 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe41100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.245+0000 7f58c6229640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78fe537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 -- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 msgr2=0x55c78fe537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78fe537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 -- 172.21.15.150:0/1756871982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 msgr2=0x55c78fe41100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe41100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 -- 172.21.15.150:0/1756871982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78fab3210 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5a28640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe41100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58c5227640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009890 tx=0x7f58b002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab0eb8b3facd5db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0039040 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f58b002c510 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58b809b550 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58b0059050 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58ca87f740 10 client.15738 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c78fb7f620 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.246+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b809b550 con 0x55c78fe3a530 2024-03-18T16:04:03.254 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f58b003f040 con 0x55c78fe3a530 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 10 client.15738 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 20 client.15738 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78fe57db0 0x55c78fe5a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 4 client.15738 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c78fdabf80 con 0x55c78fe57db0 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58c5a28640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78fe57db0 0x55c78fe5a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58c5a28640 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78fe57db0 0x55c78fe5a1e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af7a8caeef6cf8c6 server_cookie=1d638cd26fd95cb in_seq=0 out_seq=0 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.247+0000 7f58a97fa640 10 client.15738 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58a97fa640 1 -- 172.21.15.150:0/1756871982 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c78fdabf80 con 0x55c78fe57db0 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58a97fa640 10 client.15738 handle_command_reply: tid=0 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 client.15738 shutdown 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 msgr2=0x55c78fe41640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009890 tx=0x7f58b002bb70 comp rx=0 tx=0).stop 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 shutdown_connections 2024-03-18T16:04:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78fe57db0 0x55c78fe5a1e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78fe537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe41640 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58a97fa640 0 client.15738 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1756871982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe41100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 >> 172.21.15.150:0/1756871982 conn(0x55c78fe34a50 msgr2=0x55c78fe367a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 shutdown_connections 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.248+0000 7f58ca87f740 1 -- 172.21.15.150:0/1756871982 wait complete. 2024-03-18T16:04:03.256 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.249+0000 7f58ca87f740 20 client.15738 trim_cache size 0 max 16384 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 Processor -- start 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 -- start start 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe39c10 0x55c78fe415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe5faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01ada80 con 0x55c78fe3a530 2024-03-18T16:04:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01ab350 con 0x55c78fe45a30 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58ca87f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01a91c0 con 0x55c78fe39c10 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe39c10 0x55c78fe415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45988/0 (socket says 172.21.15.150:45988) 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c6229640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe5faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 -- 172.21.15.150:0/1886008114 learned_addr learned my addr 172.21.15.150:0/1886008114 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 -- 172.21.15.150:0/1886008114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe39c10 msgr2=0x55c78fe415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe39c10 0x55c78fe415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 -- 172.21.15.150:0/1886008114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 msgr2=0x55c78fe5faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe5faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 -- 172.21.15.150:0/1886008114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78fe5a860 con 0x55c78fe3a530 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c6229640 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe5faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.265+0000 7f58c5227640 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009230 tx=0x7f58b00095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ded5ecb9e223882 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ab7fe640 1 -- 172.21.15.150:0/1886008114 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0039030 con 0x55c78fe3a530 2024-03-18T16:04:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ab7fe640 1 -- 172.21.15.150:0/1886008114 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b0050070 con 0x55c78fe3a530 2024-03-18T16:04:03.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ab7fe640 1 -- 172.21.15.150:0/1886008114 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0032b10 con 0x55c78fe3a530 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 -- 172.21.15.150:0/1886008114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 msgr2=0x55c78fe537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009230 tx=0x7f58b00095b0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 -- 172.21.15.150:0/1886008114 shutdown_connections 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe45a30 0x55c78fe5faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe537b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 --2- 172.21.15.150:0/1886008114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe39c10 0x55c78fe415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 -- 172.21.15.150:0/1886008114 >> 172.21.15.150:0/1886008114 conn(0x55c78fe34a50 msgr2=0x55c78fe40ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.266+0000 7f58ca87f740 1 -- 172.21.15.150:0/1886008114 shutdown_connections 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 -- 172.21.15.150:0/1886008114 wait complete. 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 -- start start 2024-03-18T16:04:03.275 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 Processor -- start 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe4f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78febd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01a9810 con 0x55c78fe3a530 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01aac80 con 0x55c78fe39c10 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f58c01a9ee0 con 0x55c78fe45a30 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58ca87f740 10 client.0 _ll_register_callbacks cb 0x7f58b0045030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58c5227640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46000/0 (socket says 172.21.15.150:46000) 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58c5a28640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe4f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58c5227640 1 -- 172.21.15.150:0/279615937 learned_addr learned my addr 172.21.15.150:0/279615937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.267+0000 7f58c6229640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78febd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 -- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 msgr2=0x55c78febd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78febd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.276 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 -- 172.21.15.150:0/279615937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 msgr2=0x55c78fe4f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe4f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 -- 172.21.15.150:0/279615937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78fe47e20 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58c5227640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f58b002c180 tx=0x7f58b0005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9acf00d2ad81ea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0069020 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58b0050070 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f58b0003c40 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58b0074050 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58ca87f740 10 client.15744 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c78febe660 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b0050070 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.268+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f58b002ca40 con 0x55c78fe3a530 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 10 client.15744 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 10 client.15744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:03.277 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 20 client.15744 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78febecc0 0x55c78fec10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 4 client.15744 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c78fda62d0 con 0x55c78febecc0 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58c5a28640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78febecc0 0x55c78fec10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58a97fa640 10 client.15744 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:03.278 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.269+0000 7f58c5a28640 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78febecc0 0x55c78fec10e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=940469cf8b6e27da server_cookie=690a05887dbc26f2 in_seq=0 out_seq=0 2024-03-18T16:04:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.277+0000 7f58a97fa640 1 -- 172.21.15.150:0/279615937 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c78fda62d0 con 0x55c78febecc0 2024-03-18T16:04:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.277+0000 7f58a97fa640 10 client.15744 handle_command_reply: tid=0 2024-03-18T16:04:03.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.277+0000 7f58ca87f740 1 client.15744 shutdown 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 msgr2=0x55c78fe4f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f58b002c180 tx=0x7f58b0005fc0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 shutdown_connections 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55c78febecc0 0x55c78fec10e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c78fe45a30 0x55c78febd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c78fe3a530 0x55c78fe4f5a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 --2- 172.21.15.150:0/279615937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c78fe39c10 0x55c78fe4f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58a97fa640 0 client.15744 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 >> 172.21.15.150:0/279615937 conn(0x55c78fe34a50 msgr2=0x55c78fe33320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 shutdown_connections 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 1 -- 172.21.15.150:0/279615937 wait complete. 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58ca87f740 20 client.15744 trim_cache size 0 max 16384 2024-03-18T16:04:03.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 msgr2=0x7f58ac0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 0x7f58ac0a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000c00 tx=0x7f58b00323c0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 msgr2=0x7f58c019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f58b800eee0 tx=0x7f58b800c740 comp rx=0 tx=0).stop 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 shutdown_connections 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f58ac0a2a10 0x7f58ac0a4ed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58c0105680 0x7f58c019b890 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.278+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c01012f0 0x7f58c0197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.279+0000 7f58c7c8c640 1 --2- 172.21.15.150:0/2622042807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c0100930 0x7f58c0197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.279+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 >> 172.21.15.150:0/2622042807 conn(0x7f58c0077530 msgr2=0x7f58c00fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.279+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 shutdown_connections 2024-03-18T16:04:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:03.279+0000 7f58c7c8c640 1 -- 172.21.15.150:0/2622042807 wait complete. 2024-03-18T16:04:03.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 343541 11.70 MB/sec execute 48 sec latency 12.866 ms 2024-03-18T16:04:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:03 smithi191 ceph-mon[36784]: pgmap v122: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:03 smithi156 ceph-mon[35919]: pgmap v122: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:03 smithi150 ceph-mon[31946]: pgmap v122: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:04.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 345477 11.70 MB/sec execute 49 sec latency 25.663 ms 2024-03-18T16:04:05.320 INFO:tasks.workunit.client.0.smithi150.stdout: 1 347938 11.74 MB/sec execute 50 sec latency 39.810 ms 2024-03-18T16:04:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:05 smithi191 ceph-mon[36784]: pgmap v123: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:05 smithi156 ceph-mon[35919]: pgmap v123: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:05 smithi150 ceph-mon[31946]: pgmap v123: 97 pgs: 97 active+clean; 193 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 223 op/s 2024-03-18T16:04:06.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 350013 11.66 MB/sec execute 51 sec latency 57.813 ms 2024-03-18T16:04:07.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 351880 11.67 MB/sec execute 52 sec latency 17.765 ms 2024-03-18T16:04:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:07 smithi191 ceph-mon[36784]: pgmap v124: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 352 op/s 2024-03-18T16:04:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:07 smithi156 ceph-mon[35919]: pgmap v124: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 352 op/s 2024-03-18T16:04:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:07 smithi150 ceph-mon[31946]: pgmap v124: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 352 op/s 2024-03-18T16:04:08.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 354702 11.77 MB/sec execute 53 sec latency 9.632 ms 2024-03-18T16:04:09.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 356594 11.69 MB/sec execute 54 sec latency 30.137 ms 2024-03-18T16:04:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:09 smithi191 ceph-mon[36784]: pgmap v125: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-18T16:04:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:09 smithi156 ceph-mon[35919]: pgmap v125: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-18T16:04:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:09 smithi150 ceph-mon[31946]: pgmap v125: 97 pgs: 97 active+clean; 200 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 242 op/s 2024-03-18T16:04:10.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 358813 11.72 MB/sec execute 55 sec latency 9.943 ms 2024-03-18T16:04:11.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 361008 11.71 MB/sec execute 56 sec latency 80.216 ms 2024-03-18T16:04:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:11 smithi191 ceph-mon[36784]: pgmap v126: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-18T16:04:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:11 smithi156 ceph-mon[35919]: pgmap v126: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-18T16:04:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:11 smithi150 ceph-mon[31946]: pgmap v126: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-18T16:04:12.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 362704 11.66 MB/sec execute 57 sec latency 34.010 ms 2024-03-18T16:04:13.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 365564 11.77 MB/sec execute 58 sec latency 13.684 ms 2024-03-18T16:04:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:13 smithi191 ceph-mon[36784]: pgmap v127: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:13 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:13 smithi156 ceph-mon[35919]: pgmap v127: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:13 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:13 smithi150 ceph-mon[31946]: pgmap v127: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:13 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:14.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 367481 11.67 MB/sec execute 59 sec latency 19.559 ms 2024-03-18T16:04:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:14 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:14.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:14 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:14.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:14 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:04:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:14 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:14 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:14 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:04:15.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:14 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:15.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:14 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:15.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:14 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:04:15.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 369489 11.70 MB/sec execute 60 sec latency 14.843 ms 2024-03-18T16:04:15.919 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:15 smithi156 ceph-mon[35919]: pgmap v128: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:15 smithi191 ceph-mon[36784]: pgmap v128: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:15 smithi150 ceph-mon[31946]: pgmap v128: 97 pgs: 97 active+clean; 206 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 233 op/s 2024-03-18T16:04:16.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 371777 11.71 MB/sec execute 61 sec latency 99.967 ms 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 -- 172.21.15.150:0/2034582333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490107840 msgr2=0x7fc490107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 --2- 172.21.15.150:0/2034582333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490107840 0x7fc490107c20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc484009900 tx=0x7fc48402bc30 comp rx=0 tx=0).stop 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 -- 172.21.15.150:0/2034582333 shutdown_connections 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 --2- 172.21.15.150:0/2034582333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490069800 0x7fc49010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 --2- 172.21.15.150:0/2034582333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 0x7fc4900692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 --2- 172.21.15.150:0/2034582333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490107840 0x7fc490107c20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 -- 172.21.15.150:0/2034582333 >> 172.21.15.150:0/2034582333 conn(0x7fc490077530 msgr2=0x7fc490077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 -- 172.21.15.150:0/2034582333 shutdown_connections 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.369+0000 7fc496aff640 1 -- 172.21.15.150:0/2034582333 wait complete. 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 Processor -- start 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 -- start start 2024-03-18T16:04:16.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 0x7fc490197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490107840 0x7fc49019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc490111b60 con 0x7fc490069800 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901119e0 con 0x7fc490068e60 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc496aff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc490111ce0 con 0x7fc490107840 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38476/0 (socket says 172.21.15.150:38476) 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 -- 172.21.15.150:0/3704743131 learned_addr learned my addr 172.21.15.150:0/3704743131 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc49489b640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 0x7fc490197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc49509c640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490107840 0x7fc49019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 -- 172.21.15.150:0/3704743131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490107840 msgr2=0x7fc49019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490107840 0x7fc49019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 -- 172.21.15.150:0/3704743131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 msgr2=0x7fc490197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 0x7fc490197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.370+0000 7fc48ffff640 1 -- 172.21.15.150:0/3704743131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49019cf10 con 0x7fc490069800 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00e9f0 tx=0x7fc47c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc48dffb640 1 -- 172.21.15.150:0/3704743131 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c00cd80 con 0x7fc490069800 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc48dffb640 1 -- 172.21.15.150:0/3704743131 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c010910 con 0x7fc490069800 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49019d200 con 0x7fc490069800 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc48dffb640 1 -- 172.21.15.150:0/3704743131 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c018990 con 0x7fc490069800 2024-03-18T16:04:16.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.371+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4901a4a90 con 0x7fc490069800 2024-03-18T16:04:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48dffb640 1 -- 172.21.15.150:0/3704743131 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fc47c010ac0 con 0x7fc490069800 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 Processor -- start 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 -- start start 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d6953ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d695336a40 con 0x55d6953db820 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d6952df920 con 0x55d6953d2a50 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc4997c8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d695332ae0 con 0x55d6953d2130 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48dffb640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 0x7fc4800a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d6953ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc49489b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d6953ddc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38484/0 (socket says 172.21.15.150:38484) 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc49489b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:45134/0 (socket says 172.21.15.150:45134) 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48ffff640 1 -- 172.21.15.150:0/310671437 learned_addr learned my addr 172.21.15.150:0/310671437 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc49509c640 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc49509c640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 0x7fc4800a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.374+0000 7fc48dffb640 1 -- 172.21.15.150:0/3704743131 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc47c0cf0b0 con 0x7fc490069800 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 -- 172.21.15.150:0/310671437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 msgr2=0x55d6953d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 -- 172.21.15.150:0/310671437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 msgr2=0x55d6953ddc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d6953ddc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 -- 172.21.15.150:0/310671437 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6952f7460 con 0x55d6953d2a50 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49489b640 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc484000c00 tx=0x7fc484005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f15e8f88a8cc20ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc49509c640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 0x7fc4800a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc488005c40 tx=0x7fc488009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc47affd640 1 -- 172.21.15.150:0/310671437 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484042070 con 0x55d6953d2a50 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc47affd640 1 -- 172.21.15.150:0/310671437 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48402c8b0 con 0x55d6953d2a50 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.375+0000 7fc47affd640 1 -- 172.21.15.150:0/310671437 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484004c60 con 0x55d6953d2a50 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- 172.21.15.150:0/310671437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 msgr2=0x55d6953db2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc484000c00 tx=0x7fc484005be0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- 172.21.15.150:0/310671437 shutdown_connections 2024-03-18T16:04:16.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d6953ddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953db2e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 --2- 172.21.15.150:0/310671437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- 172.21.15.150:0/310671437 >> 172.21.15.150:0/310671437 conn(0x55d6953cca50 msgr2=0x55d6953cce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- 172.21.15.150:0/310671437 shutdown_connections 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- 172.21.15.150:0/310671437 wait complete. 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 -- start start 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.376+0000 7fc4997c8740 1 Processor -- start 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953d8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953db820 0x55d6953d9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d6953de3f0 con 0x55d6953d2130 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d6952f2d70 con 0x55d6953d2a50 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d6953de570 con 0x55d6953db820 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 10 client.0 _ll_register_callbacks cb 0x55d69504aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38500/0 (socket says 172.21.15.150:38500) 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 -- 172.21.15.150:0/1589069887 learned_addr learned my addr 172.21.15.150:0/1589069887 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc48ffff640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953d8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49509c640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953db820 0x55d6953d9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 -- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953db820 msgr2=0x55d6953d9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953db820 0x55d6953d9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 -- 172.21.15.150:0/1589069887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 msgr2=0x55d6953d8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953d8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 -- 172.21.15.150:0/1589069887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69504b210 con 0x55d6953d2130 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc48ffff640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953d8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc49489b640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc48402f020 tx=0x7fc48402c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24caf4ec7e2ddeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc484042040 con 0x55d6953d2130 2024-03-18T16:04:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc484047870 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.377+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc48402c8b0 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc484047a70 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d695117620 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48402c8b0 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc484058430 con 0x55d6953d2130 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 20 client.15756 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6953eb770 0x55d6953edb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 4 client.15756 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d695343f80 con 0x55d6953eb770 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.378+0000 7fc48ffff640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6953eb770 0x55d6953edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.379+0000 7fc46ffff640 10 client.15756 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.379+0000 7fc48ffff640 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6953eb770 0x55d6953edb60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56c664119a8f412a server_cookie=2af35525eee9be57 in_seq=0 out_seq=0 2024-03-18T16:04:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.379+0000 7fc46ffff640 1 -- 172.21.15.150:0/1589069887 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d695343f80 con 0x55d6953eb770 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc46ffff640 10 client.15756 handle_command_reply: tid=0 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 client.15756 shutdown 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 msgr2=0x55d6953d87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc48402f020 tx=0x7fc48402c6d0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 shutdown_connections 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6953eb770 0x55d6953edb60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953db820 0x55d6953d9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953d8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 --2- 172.21.15.150:0/1589069887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2130 0x55d6953d87e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc46ffff640 0 client.15756 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 >> 172.21.15.150:0/1589069887 conn(0x55d6953cca50 msgr2=0x55d6953ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 shutdown_connections 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 1 -- 172.21.15.150:0/1589069887 wait complete. 2024-03-18T16:04:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.380+0000 7fc4997c8740 20 client.15756 trim_cache size 0 max 16384 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 Processor -- start 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 -- start start 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953daf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953db820 0x55d6953f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901afa20 con 0x55d6953d2a50 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ad2f0 con 0x55d6953db820 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc4997c8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ab140 con 0x55d6953d2130 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc49489b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38506/0 (socket says 172.21.15.150:38506) 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc49489b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953daf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:52638/0 (socket says 172.21.15.150:52638) 2024-03-18T16:04:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc48ffff640 1 -- 172.21.15.150:0/3485323467 learned_addr learned my addr 172.21.15.150:0/3485323467 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.395+0000 7fc49509c640 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953db820 0x55d6953f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 -- 172.21.15.150:0/3485323467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 msgr2=0x55d6953daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953daf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 -- 172.21.15.150:0/3485323467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953db820 msgr2=0x55d6953f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953db820 0x55d6953f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 -- 172.21.15.150:0/3485323467 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6953cc880 con 0x55d6953d2a50 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc48ffff640 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00c6a0 tx=0x7fc47c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89abd5ba3a0c4e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc47a7fc640 1 -- 172.21.15.150:0/3485323467 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c0b4070 con 0x55d6953d2a50 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc47a7fc640 1 -- 172.21.15.150:0/3485323467 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c09f8b0 con 0x55d6953d2a50 2024-03-18T16:04:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.396+0000 7fc47a7fc640 1 -- 172.21.15.150:0/3485323467 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47c0afda0 con 0x55d6953d2a50 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- 172.21.15.150:0/3485323467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 msgr2=0x55d6953e7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00c6a0 tx=0x7fc47c086040 comp rx=0 tx=0).stop 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- 172.21.15.150:0/3485323467 shutdown_connections 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953db820 0x55d6953f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953d2a50 0x55d6953e7190 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 --2- 172.21.15.150:0/3485323467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953daf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- 172.21.15.150:0/3485323467 >> 172.21.15.150:0/3485323467 conn(0x55d6953cca50 msgr2=0x55d6953cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- 172.21.15.150:0/3485323467 shutdown_connections 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- 172.21.15.150:0/3485323467 wait complete. 2024-03-18T16:04:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 -- start start 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.397+0000 7fc4997c8740 1 Processor -- start 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953df470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901ab7b0 con 0x55d6953db820 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901acc20 con 0x55d6953d2a50 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901abe80 con 0x55d6953d2130 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 10 client.0 _ll_register_callbacks cb 0x7fc47c09b350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49489b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953df470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38514/0 (socket says 172.21.15.150:38514) 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 -- 172.21.15.150:0/411463632 learned_addr learned my addr 172.21.15.150:0/411463632 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc48ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 -- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 msgr2=0x55d6953df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953df470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 -- 172.21.15.150:0/411463632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 msgr2=0x55d6953df9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953df9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 -- 172.21.15.150:0/411463632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d695450dc0 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49489b640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc49509c640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc488003bb0 tx=0x7fc488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7136b07db7d6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48802f060 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7fc48802a650 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc47c09f8b0 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.398+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc48802a850 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 10 client.15762 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d695451f30 con 0x55d6953db820 2024-03-18T16:04:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c09f8b0 con 0x55d6953db820 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc48801d040 con 0x55d6953db820 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 10 client.15762 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 20 client.15762 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6954526e0 0x55d695454b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 4 client.15762 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d69533e2d0 con 0x55d6954526e0 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc49489b640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6954526e0 0x55d695454b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc46ffff640 10 client.15762 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.399+0000 7fc49489b640 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6954526e0 0x55d695454b00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52964efac9e02ad server_cookie=40c912c29eba3cc6 in_seq=0 out_seq=0 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc46ffff640 1 -- 172.21.15.150:0/411463632 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d69533e2d0 con 0x55d6954526e0 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc46ffff640 10 client.15762 handle_command_reply: tid=0 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 client.15762 shutdown 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 msgr2=0x55d695450880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc488003bb0 tx=0x7fc488004a40 comp rx=0 tx=0).stop 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 shutdown_connections 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d6954526e0 0x55d695454b00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d6953db820 0x55d695450880 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d6953d2a50 0x55d6953df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc46ffff640 0 client.15762 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 --2- 172.21.15.150:0/411463632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6953d2130 0x55d6953df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 >> 172.21.15.150:0/411463632 conn(0x55d6953cca50 msgr2=0x55d6953d8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 shutdown_connections 2024-03-18T16:04:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 1 -- 172.21.15.150:0/411463632 wait complete. 2024-03-18T16:04:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.408+0000 7fc4997c8740 20 client.15762 trim_cache size 0 max 16384 2024-03-18T16:04:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:04:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:04:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:04:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 msgr2=0x7fc4800a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 0x7fc4800a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc488005c40 tx=0x7fc488009500 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 msgr2=0x7fc490198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00e9f0 tx=0x7fc47c00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 shutdown_connections 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc4800a2a90 0x7fc4800a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc490107840 0x7fc49019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc490069800 0x7fc490198460 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 --2- 172.21.15.150:0/3704743131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc490068e60 0x7fc490197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 >> 172.21.15.150:0/3704743131 conn(0x7fc490077530 msgr2=0x7fc4900fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 shutdown_connections 2024-03-18T16:04:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:16.409+0000 7fc496aff640 1 -- 172.21.15.150:0/3704743131 wait complete. 2024-03-18T16:04:17.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 373368 11.65 MB/sec execute 62 sec latency 13.413 ms 2024-03-18T16:04:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:17 smithi191 ceph-mon[36784]: pgmap v129: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 348 op/s 2024-03-18T16:04:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:17 smithi156 ceph-mon[35919]: pgmap v129: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 348 op/s 2024-03-18T16:04:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:17 smithi150 ceph-mon[31946]: pgmap v129: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 348 op/s 2024-03-18T16:04:18.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 376199 11.75 MB/sec execute 63 sec latency 9.443 ms 2024-03-18T16:04:19.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 377660 11.65 MB/sec execute 64 sec latency 131.774 ms 2024-03-18T16:04:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:19 smithi191 ceph-mon[36784]: pgmap v130: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-03-18T16:04:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:19 smithi156 ceph-mon[35919]: pgmap v130: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-03-18T16:04:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:19 smithi150 ceph-mon[31946]: pgmap v130: 97 pgs: 97 active+clean; 213 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 219 op/s 2024-03-18T16:04:20.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 380066 11.69 MB/sec execute 65 sec latency 11.776 ms 2024-03-18T16:04:21.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 382122 11.63 MB/sec execute 66 sec latency 96.769 ms 2024-03-18T16:04:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:21 smithi191 ceph-mon[36784]: pgmap v131: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 314 op/s 2024-03-18T16:04:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:21 smithi156 ceph-mon[35919]: pgmap v131: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 314 op/s 2024-03-18T16:04:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:21 smithi150 ceph-mon[31946]: pgmap v131: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 314 op/s 2024-03-18T16:04:22.321 INFO:tasks.workunit.client.0.smithi150.stdout: 1 384017 11.63 MB/sec execute 67 sec latency 11.184 ms 2024-03-18T16:04:23.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 386789 11.71 MB/sec execute 68 sec latency 19.117 ms 2024-03-18T16:04:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:23 smithi191 ceph-mon[36784]: pgmap v132: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:23 smithi156 ceph-mon[35919]: pgmap v132: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:23 smithi150 ceph-mon[31946]: pgmap v132: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:24.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 388767 11.65 MB/sec execute 69 sec latency 18.847 ms 2024-03-18T16:04:25.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 390816 11.67 MB/sec execute 70 sec latency 12.380 ms 2024-03-18T16:04:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:25 smithi191 ceph-mon[36784]: pgmap v133: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:25 smithi156 ceph-mon[35919]: pgmap v133: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:25 smithi150 ceph-mon[31946]: pgmap v133: 97 pgs: 97 active+clean; 212 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 210 op/s 2024-03-18T16:04:26.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 393117 11.67 MB/sec execute 71 sec latency 80.009 ms 2024-03-18T16:04:27.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 395024 11.64 MB/sec execute 72 sec latency 12.506 ms 2024-03-18T16:04:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:27 smithi191 ceph-mon[36784]: pgmap v134: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 319 op/s 2024-03-18T16:04:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:27 smithi156 ceph-mon[35919]: pgmap v134: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 319 op/s 2024-03-18T16:04:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:27 smithi150 ceph-mon[31946]: pgmap v134: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 319 op/s 2024-03-18T16:04:28.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 397676 11.72 MB/sec execute 73 sec latency 21.964 ms 2024-03-18T16:04:29.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 400076 11.71 MB/sec execute 74 sec latency 16.784 ms 2024-03-18T16:04:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.504+0000 7f7ab1861640 1 -- 172.21.15.150:0/222118426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac101780 msgr2=0x7f7aac101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.504+0000 7f7ab1861640 1 --2- 172.21.15.150:0/222118426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac101780 0x7f7aac101c00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009930 tx=0x7f7a9402bc30 comp rx=0 tx=0).stop 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 -- 172.21.15.150:0/222118426 shutdown_connections 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 --2- 172.21.15.150:0/222118426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac102140 0x7f7aac10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 --2- 172.21.15.150:0/222118426 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac101780 0x7f7aac101c00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 --2- 172.21.15.150:0/222118426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac107790 0x7f7aac107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 -- 172.21.15.150:0/222118426 >> 172.21.15.150:0/222118426 conn(0x7f7aac0fd270 msgr2=0x7f7aac0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 -- 172.21.15.150:0/222118426 shutdown_connections 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 -- 172.21.15.150:0/222118426 wait complete. 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 Processor -- start 2024-03-18T16:04:29.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 -- start start 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.505+0000 7f7ab1861640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 0x7f7aac074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7ab1861640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 0x7f7aac074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7ab1861640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac111ba0 con 0x7f7aac102140 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7ab1861640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac111a20 con 0x7f7aac107790 2024-03-18T16:04:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7ab1861640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac111d20 con 0x7f7aac101780 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aa3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 0x7f7aac074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:32774/0 (socket says 172.21.15.150:32774) 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 -- 172.21.15.150:0/2191108617 learned_addr learned my addr 172.21.15.150:0/2191108617 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 0x7f7aac074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 -- 172.21.15.150:0/2191108617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 msgr2=0x7f7aac074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 0x7f7aac074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 -- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 msgr2=0x7f7aac074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 0x7f7aac074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 -- 172.21.15.150:0/2191108617 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aac1a65a0 con 0x7f7aac101780 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 0x7f7aac074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 0x7f7aac074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.506+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00e9f0 tx=0x7f7a9c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.507+0000 7f7ab085f640 1 -- 172.21.15.150:0/2191108617 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c00cd80 con 0x7f7aac101780 2024-03-18T16:04:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.507+0000 7f7ab085f640 1 -- 172.21.15.150:0/2191108617 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c005500 con 0x7f7aac101780 2024-03-18T16:04:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.507+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aac1a6890 con 0x7f7aac101780 2024-03-18T16:04:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.507+0000 7f7ab085f640 1 -- 172.21.15.150:0/2191108617 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c010690 con 0x7f7aac101780 2024-03-18T16:04:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.507+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aac1a6d60 con 0x7f7aac101780 2024-03-18T16:04:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 Processor -- start 2024-03-18T16:04:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 -- start start 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab085f640 1 -- 172.21.15.150:0/2191108617 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f7a9c0046c0 con 0x7f7aac101780 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea85142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea8514820 0x55fea8516c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fea846fa40 con 0x55fea8514820 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fea8418920 con 0x55fea850ba50 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7ab4455740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fea846bae0 con 0x55fea850b130 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7aaaffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea85142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:32782/0 (socket says 172.21.15.150:32782) 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 -- 172.21.15.150:0/4037484331 learned_addr learned my addr 172.21.15.150:0/4037484331 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.509+0000 7f7aa3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea8514820 0x55fea8516c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 -- 172.21.15.150:0/4037484331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 msgr2=0x55fea85142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea85142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 -- 172.21.15.150:0/4037484331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea8514820 msgr2=0x55fea8516c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea8514820 0x55fea8516c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 -- 172.21.15.150:0/4037484331 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fea8430460 con 0x55fea850b130 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea85142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7ab085f640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 0x7f7a980a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40117e0 tx=0x7f7aa4011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e823563f2f70272 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 0x7f7a980a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aa1ffb640 1 -- 172.21.15.150:0/4037484331 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4011310 con 0x55fea850b130 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aa1ffb640 1 -- 172.21.15.150:0/4037484331 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4003e50 con 0x55fea850b130 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7aa1ffb640 1 -- 172.21.15.150:0/4037484331 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4004560 con 0x55fea850b130 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.510+0000 7f7ab085f640 1 -- 172.21.15.150:0/2191108617 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7a9c0cf1f0 con 0x7f7aac101780 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- 172.21.15.150:0/4037484331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 msgr2=0x55fea850b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7aaaffd640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 0x7f7a980a4ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0cca40 tx=0x7f7a9c0d2d40 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40117e0 tx=0x7f7aa4011ca0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- 172.21.15.150:0/4037484331 shutdown_connections 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea8514820 0x55fea8516c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea85142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 --2- 172.21.15.150:0/4037484331 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850b130 0x55fea850b510 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- 172.21.15.150:0/4037484331 >> 172.21.15.150:0/4037484331 conn(0x55fea8505a50 msgr2=0x55fea8505e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- 172.21.15.150:0/4037484331 shutdown_connections 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- 172.21.15.150:0/4037484331 wait complete. 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.511+0000 7f7ab4455740 1 -- start start 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 Processor -- start 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea85118c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea8511e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8512340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fea8517450 con 0x55fea850b130 2024-03-18T16:04:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fea842bd70 con 0x55fea8514820 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fea85175d0 con 0x55fea850ba50 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7ab4455740 10 client.0 _ll_register_callbacks cb 0x55fea8183a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea85118c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea8511e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea85118c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41626/0 (socket says 172.21.15.150:41626) 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aab7fe640 1 -- 172.21.15.150:0/3731919863 learned_addr learned my addr 172.21.15.150:0/3731919863 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8512340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 -- 172.21.15.150:0/3731919863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 msgr2=0x55fea8512340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8512340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 -- 172.21.15.150:0/3731919863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 msgr2=0x55fea85118c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea85118c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aa3fff640 1 -- 172.21.15.150:0/3731919863 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fea8184210 con 0x55fea850ba50 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.512+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8512340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:04:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea8511e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009230 tx=0x7f7a9402c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bcefade1434ce68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402ccb0 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7a94004040 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa4003e50 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a94004240 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 10 client.25597 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fea8250620 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4003e50 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f7a94003860 con 0x55fea850ba50 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 10 client.25597 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 20 client.25597 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea85249b0 0x55fea8526da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 4 client.25597 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.513+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fea847cf80 con 0x55fea85249b0 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.514+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea85249b0 0x55fea8526da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.514+0000 7f7a8affd640 10 client.25597 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.514+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea85249b0 0x55fea8526da0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74481eff04b7323b server_cookie=e783398b1264413 in_seq=0 out_seq=0 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.514+0000 7f7a8affd640 1 -- 172.21.15.150:0/3731919863 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fea847cf80 con 0x55fea85249b0 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.514+0000 7f7a8affd640 10 client.25597 handle_command_reply: tid=0 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 client.25597 shutdown 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 msgr2=0x55fea8511e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea8511e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009230 tx=0x7f7a9402c880 comp rx=0 tx=0).stop 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 shutdown_connections 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea85249b0 0x55fea8526da0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8512340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea8511e00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7a8affd640 0 client.25597 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 --2- 172.21.15.150:0/3731919863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea85118c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 >> 172.21.15.150:0/3731919863 conn(0x55fea8505a50 msgr2=0x55fea85077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 shutdown_connections 2024-03-18T16:04:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 1 -- 172.21.15.150:0/3731919863 wait complete. 2024-03-18T16:04:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.515+0000 7f7ab4455740 20 client.25597 trim_cache size 0 max 16384 2024-03-18T16:04:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.531+0000 7f7ab4455740 1 Processor -- start 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.531+0000 7f7ab4455740 1 -- start start 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.531+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.531+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea85201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.531+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8504a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7ab4455740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1b1eb0 con 0x55fea850b130 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7ab4455740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1af780 con 0x55fea8514820 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7ab4455740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ad5d0 con 0x55fea850ba50 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41632/0 (socket says 172.21.15.150:41632) 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 -- 172.21.15.150:0/2526516060 learned_addr learned my addr 172.21.15.150:0/2526516060 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea85201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8504a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 -- 172.21.15.150:0/2526516060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 msgr2=0x55fea85201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea85201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 -- 172.21.15.150:0/2526516060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 msgr2=0x55fea8504a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8504a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 -- 172.21.15.150:0/2526516060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fea8501310 con 0x55fea850b130 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4014040 tx=0x7f7aa4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03e81100e1fd44f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aa17fa640 1 -- 172.21.15.150:0/2526516060 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4023040 con 0x55fea850b130 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aa17fa640 1 -- 172.21.15.150:0/2526516060 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4011490 con 0x55fea850b130 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.532+0000 7f7aa17fa640 1 -- 172.21.15.150:0/2526516060 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40057a0 con 0x55fea850b130 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- 172.21.15.150:0/2526516060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 msgr2=0x55fea8513ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4014040 tx=0x7f7aa4003970 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- 172.21.15.150:0/2526516060 shutdown_connections 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea8514820 0x55fea8504a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea850ba50 0x55fea85201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 --2- 172.21.15.150:0/2526516060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8513ff0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- 172.21.15.150:0/2526516060 >> 172.21.15.150:0/2526516060 conn(0x55fea8505a50 msgr2=0x55fea8504320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- 172.21.15.150:0/2526516060 shutdown_connections 2024-03-18T16:04:29.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- 172.21.15.150:0/2526516060 wait complete. 2024-03-18T16:04:29.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.533+0000 7f7ab4455740 1 -- start start 2024-03-18T16:04:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 Processor -- start 2024-03-18T16:04:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8518950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea8514820 0x55fea8583c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1adc40 con 0x55fea850b130 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1af0b0 con 0x55fea850ba50 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ae310 con 0x55fea8514820 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7ab4455740 10 client.0 _ll_register_callbacks cb 0x7f7a94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8518950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8518950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41636/0 (socket says 172.21.15.150:41636) 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aab7fe640 1 -- 172.21.15.150:0/505877056 learned_addr learned my addr 172.21.15.150:0/505877056 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:60120/0 (socket says 172.21.15.150:60120) 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea8514820 0x55fea8583c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 -- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea8514820 msgr2=0x55fea8583c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea8514820 0x55fea8583c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 -- 172.21.15.150:0/505877056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 msgr2=0x55fea8518950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:29.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8518950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.534+0000 7f7aa3fff640 1 -- 172.21.15.150:0/505877056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fea8584230 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7aa3fff640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55fea8512e90 tx=0x7f7a94005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a9e501614f6ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94005d50 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f7a940035f0 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fea84772d0 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a94061050 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 10 client.25550 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fea858b5c0 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fea84772d0 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f7a940484c0 con 0x55fea850ba50 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 10 client.25550 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 10 client.25550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:29.541 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 20 client.25550 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea858bdc0 0x55fea858e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 4 client.25550 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.535+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fea83f5770 con 0x55fea858bdc0 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.536+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea858bdc0 0x55fea858e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.536+0000 7f7a8affd640 10 client.25550 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:29.542 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.536+0000 7f7aab7fe640 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea858bdc0 0x55fea858e1e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da29da407ae4685 server_cookie=53dc72c50c755890 in_seq=0 out_seq=0 2024-03-18T16:04:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7a8affd640 1 -- 172.21.15.150:0/505877056 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fea83f5770 con 0x55fea858bdc0 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7a8affd640 10 client.25550 handle_command_reply: tid=0 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 client.25550 shutdown 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 msgr2=0x55fea8518e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x55fea8512e90 tx=0x7f7a94005bc0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 shutdown_connections 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fea858bdc0 0x55fea858e1e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fea8514820 0x55fea8583c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fea850ba50 0x55fea8518e90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 --2- 172.21.15.150:0/505877056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea850b130 0x55fea8518950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7a8affd640 0 client.25550 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 >> 172.21.15.150:0/505877056 conn(0x55fea8505a50 msgr2=0x55fea8504bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 shutdown_connections 2024-03-18T16:04:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 1 -- 172.21.15.150:0/505877056 wait complete. 2024-03-18T16:04:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab4455740 20 client.25550 trim_cache size 0 max 16384 2024-03-18T16:04:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:04:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:04:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:04:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 msgr2=0x7f7a980a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 0x7f7a980a4ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0cca40 tx=0x7f7a9c0d2d40 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 msgr2=0x7f7aac073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.545+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00e9f0 tx=0x7f7a9c00edf0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 shutdown_connections 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7a980a2a10 0x7f7a980a4ed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aac107790 0x7f7aac074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac102140 0x7f7aac074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 --2- 172.21.15.150:0/2191108617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7aac101780 0x7f7aac073ae0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 >> 172.21.15.150:0/2191108617 conn(0x7f7aac0fd270 msgr2=0x7f7aac0fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 shutdown_connections 2024-03-18T16:04:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:29.546+0000 7f7ab1861640 1 -- 172.21.15.150:0/2191108617 wait complete. 2024-03-18T16:04:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:29 smithi191 ceph-mon[36784]: pgmap v135: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-03-18T16:04:30.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:29 smithi156 ceph-mon[35919]: pgmap v135: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-03-18T16:04:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:29 smithi150 ceph-mon[31946]: pgmap v135: 97 pgs: 97 active+clean; 219 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 204 op/s 2024-03-18T16:04:30.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 401838 11.67 MB/sec execute 75 sec latency 22.019 ms 2024-03-18T16:04:31.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 404534 11.74 MB/sec execute 76 sec latency 73.698 ms 2024-03-18T16:04:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:31 smithi191 ceph-mon[36784]: pgmap v136: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-03-18T16:04:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:31 smithi156 ceph-mon[35919]: pgmap v136: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-03-18T16:04:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:31 smithi150 ceph-mon[31946]: pgmap v136: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 312 op/s 2024-03-18T16:04:32.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 406747 11.69 MB/sec execute 77 sec latency 16.838 ms 2024-03-18T16:04:33.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 408632 11.70 MB/sec execute 78 sec latency 15.630 ms 2024-03-18T16:04:33.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:33 smithi191 ceph-mon[36784]: pgmap v137: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:33 smithi156 ceph-mon[35919]: pgmap v137: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:33 smithi150 ceph-mon[31946]: pgmap v137: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:34.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 411310 11.73 MB/sec execute 79 sec latency 17.288 ms 2024-03-18T16:04:35.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 413150 11.71 MB/sec execute 80 sec latency 12.593 ms 2024-03-18T16:04:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:35 smithi191 ceph-mon[36784]: pgmap v138: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:35 smithi156 ceph-mon[35919]: pgmap v138: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:35 smithi150 ceph-mon[31946]: pgmap v138: 97 pgs: 97 active+clean; 225 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:36.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 415396 11.74 MB/sec execute 81 sec latency 99.171 ms 2024-03-18T16:04:37.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 417531 11.69 MB/sec execute 82 sec latency 26.324 ms 2024-03-18T16:04:37.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:37 smithi191 ceph-mon[36784]: pgmap v139: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-03-18T16:04:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:37 smithi156 ceph-mon[35919]: pgmap v139: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-03-18T16:04:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:37 smithi150 ceph-mon[31946]: pgmap v139: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 314 op/s 2024-03-18T16:04:38.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 419410 11.69 MB/sec execute 83 sec latency 15.981 ms 2024-03-18T16:04:39.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 422238 11.76 MB/sec execute 84 sec latency 18.107 ms 2024-03-18T16:04:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:39 smithi191 ceph-mon[36784]: pgmap v140: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-03-18T16:04:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:39 smithi156 ceph-mon[35919]: pgmap v140: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-03-18T16:04:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:39 smithi150 ceph-mon[31946]: pgmap v140: 97 pgs: 97 active+clean; 232 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 205 op/s 2024-03-18T16:04:40.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 424274 11.70 MB/sec execute 85 sec latency 20.760 ms 2024-03-18T16:04:41.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 426116 11.72 MB/sec execute 86 sec latency 71.230 ms 2024-03-18T16:04:41.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:41 smithi191 ceph-mon[36784]: pgmap v141: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-18T16:04:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:41 smithi156 ceph-mon[35919]: pgmap v141: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-18T16:04:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:41 smithi150 ceph-mon[31946]: pgmap v141: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-18T16:04:42.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 428603 11.73 MB/sec execute 87 sec latency 25.920 ms 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- 172.21.15.150:0/314234522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de41077e0 msgr2=0x7f8de4107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 --2- 172.21.15.150:0/314234522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de41077e0 0x7f8de4107bc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009930 tx=0x7f8dd402bc30 comp rx=0 tx=0).stop 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- 172.21.15.150:0/314234522 shutdown_connections 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 --2- 172.21.15.150:0/314234522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de4102190 0x7f8de410e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 --2- 172.21.15.150:0/314234522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41017d0 0x7f8de4101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 --2- 172.21.15.150:0/314234522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de41077e0 0x7f8de4107bc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- 172.21.15.150:0/314234522 >> 172.21.15.150:0/314234522 conn(0x7f8de40fd2a0 msgr2=0x7f8de40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- 172.21.15.150:0/314234522 shutdown_connections 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- 172.21.15.150:0/314234522 wait complete. 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 Processor -- start 2024-03-18T16:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.635+0000 7f8dea976640 1 -- start start 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de41017d0 0x7f8de4073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41077e0 0x7f8de4074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111bf0 con 0x7f8de4102190 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111a70 con 0x7f8de41017d0 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8dea976640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111d70 con 0x7f8de41077e0 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39832/0 (socket says 172.21.15.150:39832) 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 -- 172.21.15.150:0/1518228691 learned_addr learned my addr 172.21.15.150:0/1518228691 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de3fff640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de41017d0 0x7f8de4073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de8f13640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41077e0 0x7f8de4074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 -- 172.21.15.150:0/1518228691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41077e0 msgr2=0x7f8de4074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41077e0 0x7f8de4074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 -- 172.21.15.150:0/1518228691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de41017d0 msgr2=0x7f8de4073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de41017d0 0x7f8de4073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 -- 172.21.15.150:0/1518228691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de41a6530 con 0x7f8de4102190 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.636+0000 7f8de37fe640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800e9f0 tx=0x7f8dd800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.637+0000 7f8de17fa640 1 -- 172.21.15.150:0/1518228691 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800cd80 con 0x7f8de4102190 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.637+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de41a6820 con 0x7f8de4102190 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.637+0000 7f8de17fa640 1 -- 172.21.15.150:0/1518228691 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd8005500 con 0x7f8de4102190 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.637+0000 7f8de17fa640 1 -- 172.21.15.150:0/1518228691 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8010690 con 0x7f8de4102190 2024-03-18T16:04:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.637+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de41a6d70 con 0x7f8de4102190 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8de17fa640 1 -- 172.21.15.150:0/1518228691 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f8dd80046c0 con 0x7f8de4102190 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 Processor -- start 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 -- start start 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aece12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8de17fa640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 0x7f8dcc0a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aece1820 0x5590aece3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5590aec3ca40 con 0x5590aece1820 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5590aebe5920 con 0x5590aecd8130 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.639+0000 7f8ded634740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5590aec38ae0 con 0x5590aecd8a50 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de8f13640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aece12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de8f13640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:56072/0 (socket says 172.21.15.150:56072) 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de8f13640 1 -- 172.21.15.150:0/4020384217 learned_addr learned my addr 172.21.15.150:0/4020384217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de37fe640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 0x7f8dcc0a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de8f13640 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aece1820 0x5590aece3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de17fa640 1 -- 172.21.15.150:0/1518228691 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8dd80cf1f0 con 0x7f8de4102190 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 -- 172.21.15.150:0/4020384217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 msgr2=0x5590aecd8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 -- 172.21.15.150:0/4020384217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aece1820 msgr2=0x5590aece3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aece1820 0x5590aece3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 -- 172.21.15.150:0/4020384217 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590aebfd460 con 0x5590aecd8a50 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de8f13640 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de3fff640 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aece12e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009230 tx=0x7f8dd4009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e24b423ceca9d53e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8de37fe640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 0x7f8dcc0a4f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8dd809b5a0 tx=0x7f8dd806d590 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8dca7fc640 1 -- 172.21.15.150:0/4020384217 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4042070 con 0x5590aecd8a50 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8dca7fc640 1 -- 172.21.15.150:0/4020384217 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd402c690 con 0x5590aecd8a50 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.640+0000 7f8dca7fc640 1 -- 172.21.15.150:0/4020384217 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4004af0 con 0x5590aecd8a50 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 -- 172.21.15.150:0/4020384217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 msgr2=0x5590aece12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aece12e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009230 tx=0x7f8dd4009610 comp rx=0 tx=0).stop 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 -- 172.21.15.150:0/4020384217 shutdown_connections 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aece1820 0x5590aece3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aece12e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 --2- 172.21.15.150:0/4020384217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aecd8130 0x5590aecd8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 -- 172.21.15.150:0/4020384217 >> 172.21.15.150:0/4020384217 conn(0x5590aecd2a50 msgr2=0x5590aecd2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 -- 172.21.15.150:0/4020384217 shutdown_connections 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.641+0000 7f8ded634740 1 -- 172.21.15.150:0/4020384217 wait complete. 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 -- start start 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 Processor -- start 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aecde8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecdf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5590aece43f0 con 0x5590aecd8a50 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5590aebf8d70 con 0x5590aece1820 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5590aece4570 con 0x5590aecd8130 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8ded634740 10 client.0 _ll_register_callbacks cb 0x5590ae950a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8de8f13640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aecde8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39862/0 (socket says 172.21.15.150:39862) 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8de3fff640 1 -- 172.21.15.150:0/1265445129 learned_addr learned my addr 172.21.15.150:0/1265445129 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.642+0000 7f8de8f13640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecdf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 -- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 msgr2=0x5590aecde8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aecde8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 -- 172.21.15.150:0/1265445129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 msgr2=0x5590aecdf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecdf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 -- 172.21.15.150:0/1265445129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ae951210 con 0x5590aecd8a50 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8de3fff640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005ab0 tx=0x7f8dd402c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a98c4d0fcfb738 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4042040 con 0x5590aecd8a50 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8dd4004040 con 0x5590aecd8a50 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5590aec49f80 con 0x5590aecd8a50 2024-03-18T16:04:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8dd4004d40 con 0x5590aecd8a50 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590aea1d620 con 0x5590aecd8a50 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5590aec49f80 con 0x5590aecd8a50 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f8dd4003710 con 0x5590aecd8a50 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 20 client.15792 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aecf1860 0x5590aecf3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 4 client.15792 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.643+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590aec442d0 con 0x5590aecf1860 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.644+0000 7f8de8f13640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aecf1860 0x5590aecf3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.644+0000 7f8dc37fe640 10 client.15792 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.644+0000 7f8de8f13640 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aecf1860 0x5590aecf3c50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a98402538d5c21b5 server_cookie=5e99ab2c7b04a7ff in_seq=0 out_seq=0 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8dc37fe640 1 -- 172.21.15.150:0/1265445129 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590aec442d0 con 0x5590aecf1860 2024-03-18T16:04:42.652 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8dc37fe640 10 client.15792 handle_command_reply: tid=0 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 client.15792 shutdown 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 msgr2=0x5590aecdee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005ab0 tx=0x7f8dd402c330 comp rx=0 tx=0).stop 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 shutdown_connections 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aecf1860 0x5590aecf3c50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecdf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecdee30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 --2- 172.21.15.150:0/1265445129 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aecde8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8dc37fe640 0 client.15792 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 >> 172.21.15.150:0/1265445129 conn(0x5590aecd2a50 msgr2=0x5590aecd47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 shutdown_connections 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 1 -- 172.21.15.150:0/1265445129 wait complete. 2024-03-18T16:04:42.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.645+0000 7f8ded634740 20 client.15792 trim_cache size 0 max 16384 2024-03-18T16:04:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.660+0000 7f8ded634740 1 Processor -- start 2024-03-18T16:04:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.660+0000 7f8ded634740 1 -- start start 2024-03-18T16:04:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aece1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41b1e10 con 0x5590aecd8a50 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41af6e0 con 0x5590aece1820 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8ded634740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41ad530 con 0x5590aecd8130 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39872/0 (socket says 172.21.15.150:39872) 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 -- 172.21.15.150:0/2424386925 learned_addr learned my addr 172.21.15.150:0/2424386925 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aece1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 -- 172.21.15.150:0/2424386925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 msgr2=0x5590aece1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aece1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 -- 172.21.15.150:0/2424386925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 msgr2=0x5590aecf6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 -- 172.21.15.150:0/2424386925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590aece4440 con 0x5590aecd8a50 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8dd402c170 tx=0x7f8dd4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc9c28ae6e3212d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8dc9ffb640 1 -- 172.21.15.150:0/2424386925 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd405c070 con 0x5590aecd8a50 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8dc9ffb640 1 -- 172.21.15.150:0/2424386925 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40046c0 con 0x5590aecd8a50 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.661+0000 7f8dc9ffb640 1 -- 172.21.15.150:0/2424386925 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4042040 con 0x5590aecd8a50 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- 172.21.15.150:0/2424386925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 msgr2=0x5590aecd45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8dd402c170 tx=0x7f8dd4009610 comp rx=0 tx=0).stop 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- 172.21.15.150:0/2424386925 shutdown_connections 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8a50 0x5590aecd45b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 --2- 172.21.15.150:0/2424386925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8130 0x5590aece1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- 172.21.15.150:0/2424386925 >> 172.21.15.150:0/2424386925 conn(0x5590aecd2a50 msgr2=0x5590aecd3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- 172.21.15.150:0/2424386925 shutdown_connections 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- 172.21.15.150:0/2424386925 wait complete. 2024-03-18T16:04:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.662+0000 7f8ded634740 1 -- start start 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 Processor -- start 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aecf4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41adba0 con 0x5590aecd8130 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41af010 con 0x5590aece1820 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41ae270 con 0x5590aecd8a50 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8ded634740 10 client.0 _ll_register_callbacks cb 0x7f8ddc0115d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39888/0 (socket says 172.21.15.150:39888) 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 -- 172.21.15.150:0/2697177235 learned_addr learned my addr 172.21.15.150:0/2697177235 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aecf4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 -- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 msgr2=0x5590aecf4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aecf4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 -- 172.21.15.150:0/2697177235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 msgr2=0x5590aecf51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 -- 172.21.15.150:0/2697177235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590aed56a00 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8de8f13640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc003b70 tx=0x7f8dd4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8568eb4840bc2f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8dc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc005510 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.663+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f8ddc0056b0 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dd40046c0 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8ddc037050 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 10 client.15798 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590aed57c80 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40046c0 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f8ddc022460 con 0x5590aecd8130 2024-03-18T16:04:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 10 client.15798 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 20 client.15798 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aed583d0 0x5590aed5a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 4 client.15798 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590aebc2770 con 0x5590aed583d0 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.664+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aed583d0 0x5590aed5a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.665+0000 7f8de3fff640 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aed583d0 0x5590aed5a7f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=405a17f5344e946d server_cookie=c4fdd83957090607 in_seq=0 out_seq=0 2024-03-18T16:04:42.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.665+0000 7f8dc37fe640 10 client.15798 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8dc37fe640 1 -- 172.21.15.150:0/2697177235 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590aebc2770 con 0x5590aed583d0 2024-03-18T16:04:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8dc37fe640 10 client.15798 handle_command_reply: tid=0 2024-03-18T16:04:42.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 client.15798 shutdown 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 msgr2=0x5590aecf4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc003b70 tx=0x7f8dd4005b50 comp rx=0 tx=0).stop 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 shutdown_connections 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590aed583d0 0x5590aed5a7f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590aece1820 0x5590aecf51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590aecd8a50 0x5590aecf4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8dc37fe640 0 client.15798 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 --2- 172.21.15.150:0/2697177235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590aecd8130 0x5590aecf4750 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.673+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 >> 172.21.15.150:0/2697177235 conn(0x5590aecd2a50 msgr2=0x5590aecd1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 shutdown_connections 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8ded634740 1 -- 172.21.15.150:0/2697177235 wait complete. 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8ded634740 20 client.15798 trim_cache size 0 max 16384 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:04:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 msgr2=0x7f8dcc0a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 0x7f8dcc0a4f00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8dd809b5a0 tx=0x7f8dd806d590 comp rx=0 tx=0).stop 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 msgr2=0x7f8de4074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800e9f0 tx=0x7f8dd800edf0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 shutdown_connections 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8dcc0a2a40 0x7f8dcc0a4f00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8de41077e0 0x7f8de4074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de4102190 0x7f8de4074020 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 --2- 172.21.15.150:0/1518228691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de41017d0 0x7f8de4073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 >> 172.21.15.150:0/1518228691 conn(0x7f8de40fd2a0 msgr2=0x7f8de40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 shutdown_connections 2024-03-18T16:04:42.679 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:42.674+0000 7f8dea976640 1 -- 172.21.15.150:0/1518228691 wait complete. 2024-03-18T16:04:43.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 430429 11.70 MB/sec execute 88 sec latency 19.665 ms 2024-03-18T16:04:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:43 smithi191 ceph-mon[36784]: pgmap v142: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:43 smithi156 ceph-mon[35919]: pgmap v142: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:43 smithi150 ceph-mon[31946]: pgmap v142: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:44.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 432984 11.75 MB/sec execute 89 sec latency 62.041 ms 2024-03-18T16:04:45.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 435161 11.70 MB/sec execute 90 sec latency 12.457 ms 2024-03-18T16:04:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:45 smithi191 ceph-mon[36784]: pgmap v143: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:45 smithi156 ceph-mon[35919]: pgmap v143: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:45 smithi150 ceph-mon[31946]: pgmap v143: 97 pgs: 97 active+clean; 238 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 203 op/s 2024-03-18T16:04:46.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 436787 11.71 MB/sec execute 91 sec latency 81.731 ms 2024-03-18T16:04:47.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 439190 11.70 MB/sec execute 92 sec latency 53.999 ms 2024-03-18T16:04:47.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:47 smithi191 ceph-mon[36784]: pgmap v144: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-03-18T16:04:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:47 smithi156 ceph-mon[35919]: pgmap v144: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-03-18T16:04:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:47 smithi150 ceph-mon[31946]: pgmap v144: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 313 op/s 2024-03-18T16:04:48.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 440914 11.68 MB/sec execute 93 sec latency 19.069 ms 2024-03-18T16:04:49.322 INFO:tasks.workunit.client.0.smithi150.stdout: 1 443604 11.74 MB/sec execute 94 sec latency 19.488 ms 2024-03-18T16:04:49.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:49 smithi191 ceph-mon[36784]: pgmap v145: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:49 smithi156 ceph-mon[35919]: pgmap v145: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:49 smithi150 ceph-mon[31946]: pgmap v145: 97 pgs: 97 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:50.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 445805 11.69 MB/sec execute 95 sec latency 19.634 ms 2024-03-18T16:04:51.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 447625 11.70 MB/sec execute 96 sec latency 28.323 ms 2024-03-18T16:04:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:51 smithi191 ceph-mon[36784]: pgmap v146: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-03-18T16:04:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:51 smithi156 ceph-mon[35919]: pgmap v146: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-03-18T16:04:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:51 smithi150 ceph-mon[31946]: pgmap v146: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-03-18T16:04:52.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 449875 11.71 MB/sec execute 97 sec latency 68.885 ms 2024-03-18T16:04:53.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 451674 11.67 MB/sec execute 98 sec latency 19.522 ms 2024-03-18T16:04:53.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:53 smithi191 ceph-mon[36784]: pgmap v147: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:53 smithi156 ceph-mon[35919]: pgmap v147: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:53 smithi150 ceph-mon[31946]: pgmap v147: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:54.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 454300 11.73 MB/sec execute 99 sec latency 59.187 ms 2024-03-18T16:04:55.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 456463 11.68 MB/sec execute 100 sec latency 19.102 ms 2024-03-18T16:04:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 -- 172.21.15.150:0/4280548595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 msgr2=0x7f17e8100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 --2- 172.21.15.150:0/4280548595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8100db0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009900 tx=0x7f17d402bc30 comp rx=0 tx=0).stop 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 -- 172.21.15.150:0/4280548595 shutdown_connections 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 --2- 172.21.15.150:0/4280548595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e81012f0 0x7f17e810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 --2- 172.21.15.150:0/4280548595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8100db0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 --2- 172.21.15.150:0/4280548595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e8105680 0x7f17e8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 -- 172.21.15.150:0/4280548595 >> 172.21.15.150:0/4280548595 conn(0x7f17e8077530 msgr2=0x7f17e8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.765+0000 7f17ef122640 1 -- 172.21.15.150:0/4280548595 shutdown_connections 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 -- 172.21.15.150:0/4280548595 wait complete. 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 Processor -- start 2024-03-18T16:04:55.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 -- start start 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 0x7f17e8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8105680 0x7f17e819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111cc0 con 0x7f17e8100930 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111b40 con 0x7f17e81012f0 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ef122640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8111e40 con 0x7f17e8105680 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59996/0 (socket says 172.21.15.150:59996) 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ecebe640 1 -- 172.21.15.150:0/2163019713 learned_addr learned my addr 172.21.15.150:0/2163019713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8105680 0x7f17e819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.766+0000 7f17e7fff640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 0x7f17e8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 -- 172.21.15.150:0/2163019713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8105680 msgr2=0x7f17e819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8105680 0x7f17e819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 -- 172.21.15.150:0/2163019713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 msgr2=0x7f17e8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 0x7f17e8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 -- 172.21.15.150:0/2163019713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e819d0e0 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17e7fff640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 0x7f17e8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00e9f0 tx=0x7f17dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17e5ffb640 1 -- 172.21.15.150:0/2163019713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc00cd80 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17e5ffb640 1 -- 172.21.15.150:0/2163019713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc010910 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17e5ffb640 1 -- 172.21.15.150:0/2163019713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc018990 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e819d3d0 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.767+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e81a4c60 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e5ffb640 1 -- 172.21.15.150:0/2163019713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f17dc010ac0 con 0x7f17e8100930 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 Processor -- start 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 -- start start 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e5ffb640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 0x7f17d80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 0x5618b1612510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 0x5618b161dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5618b1576a40 con 0x5618b1612a50 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5618b151f920 con 0x5618b1612130 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17f1dee740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5618b1572ae0 con 0x5618b161b820 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 0x7f17d80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e5ffb640 1 -- 172.21.15.150:0/2163019713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f17dc0cf0d0 con 0x7f17e8100930 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60008/0 (socket says 172.21.15.150:60008) 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17e7fff640 1 -- 172.21.15.150:0/790947953 learned_addr learned my addr 172.21.15.150:0/790947953 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17ecebe640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 0x5618b1612510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.770+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 0x5618b161dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 -- 172.21.15.150:0/790947953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 msgr2=0x5618b161dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 0x5618b161dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 -- 172.21.15.150:0/790947953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 msgr2=0x5618b161b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 -- 172.21.15.150:0/790947953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b1537460 con 0x5618b1612130 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 0x5618b161dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17e7fff640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ecebe640 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 0x5618b1612510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cc870 tx=0x7f17dc09b250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9582e44832a97b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 0x7f17d80a4ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x5618b160de70 tx=0x7f17e0009b00 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17d2ffd640 1 -- 172.21.15.150:0/790947953 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc09b830 con 0x5618b1612130 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17d2ffd640 1 -- 172.21.15.150:0/790947953 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc09cab0 con 0x5618b1612130 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.771+0000 7f17d2ffd640 1 -- 172.21.15.150:0/790947953 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0a0fd0 con 0x5618b1612130 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- 172.21.15.150:0/790947953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 msgr2=0x5618b1612510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 0x5618b1612510 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cc870 tx=0x7f17dc09b250 comp rx=0 tx=0).stop 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- 172.21.15.150:0/790947953 shutdown_connections 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b161b820 0x5618b161dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b1612a50 0x5618b161b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 --2- 172.21.15.150:0/790947953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612130 0x5618b1612510 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- 172.21.15.150:0/790947953 >> 172.21.15.150:0/790947953 conn(0x5618b160ca50 msgr2=0x5618b160ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- 172.21.15.150:0/790947953 shutdown_connections 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- 172.21.15.150:0/790947953 wait complete. 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 -- start start 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.772+0000 7f17f1dee740 1 Processor -- start 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1618e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5618b161e450 con 0x5618b161b820 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5618b1532d70 con 0x5618b1612a50 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5618b161e5d0 con 0x5618b1612130 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 10 client.0 _ll_register_callbacks cb 0x5618b128aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ed6bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:49996/0 (socket says 172.21.15.150:49996) 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 -- 172.21.15.150:0/92766591 learned_addr learned my addr 172.21.15.150:0/92766591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17e7fff640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1618e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60020/0 (socket says 172.21.15.150:60020) 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 -- 172.21.15.150:0/92766591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 msgr2=0x5618b1618e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1618e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 -- 172.21.15.150:0/92766591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 msgr2=0x5618b16193a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 -- 172.21.15.150:0/92766591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b128b210 con 0x5618b1612130 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17e7fff640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1618e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17ecebe640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cc920 tx=0x7f17dc0a0a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf2fb6f94e6ff5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0b3030 con 0x5618b1612130 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17dc09cab0 con 0x5618b1612130 2024-03-18T16:04:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.773+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f17dc0860e0 con 0x5618b1612130 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17dc0a5030 con 0x5618b1612130 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 10 client.25633 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618b1357620 con 0x5618b1612130 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc09cab0 con 0x5618b1612130 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f17dc0a3a90 con 0x5618b1612130 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 10 client.25633 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 10 client.25633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 20 client.25633 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b162b910 0x5618b162dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 4 client.25633 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618b1583f80 con 0x5618b162b910 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.774+0000 7f17e7fff640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b162b910 0x5618b162dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.775+0000 7f17c7fff640 10 client.25633 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.775+0000 7f17e7fff640 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b162b910 0x5618b162dd00 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef76c2f0fe19746e server_cookie=34d7ff51ce1a0dc in_seq=0 out_seq=0 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.775+0000 7f17c7fff640 1 -- 172.21.15.150:0/92766591 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618b1583f80 con 0x5618b162b910 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.775+0000 7f17c7fff640 10 client.25633 handle_command_reply: tid=0 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.775+0000 7f17f1dee740 1 client.25633 shutdown 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 msgr2=0x5618b1618920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cc920 tx=0x7f17dc0a0a80 comp rx=0 tx=0).stop 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 shutdown_connections 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b162b910 0x5618b162dd00 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b16193a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1618e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17c7fff640 0 client.25633 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 --2- 172.21.15.150:0/92766591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1618920 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 >> 172.21.15.150:0/92766591 conn(0x5618b160ca50 msgr2=0x5618b160e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 shutdown_connections 2024-03-18T16:04:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 1 -- 172.21.15.150:0/92766591 wait complete. 2024-03-18T16:04:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.776+0000 7f17f1dee740 20 client.25633 trim_cache size 0 max 16384 2024-03-18T16:04:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 Processor -- start 2024-03-18T16:04:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 -- start start 2024-03-18T16:04:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b16272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81afc70 con 0x5618b161b820 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81ad540 con 0x5618b1612a50 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.791+0000 7f17f1dee740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81ab390 con 0x5618b1612130 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b16272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.803 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60026/0 (socket says 172.21.15.150:60026) 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ecebe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b16272f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:50012/0 (socket says 172.21.15.150:50012) 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:45324/0 (socket says 172.21.15.150:45324) 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 -- 172.21.15.150:0/742342423 learned_addr learned my addr 172.21.15.150:0/742342423 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 -- 172.21.15.150:0/742342423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 msgr2=0x5618b16272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b16272f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 -- 172.21.15.150:0/742342423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 msgr2=0x5618b1630d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 -- 172.21.15.150:0/742342423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b16118e0 con 0x5618b161b820 2024-03-18T16:04:55.804 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17e7fff640 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f17e00131d0 tx=0x7f17e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf719bf719a9dfac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17d27fc640 1 -- 172.21.15.150:0/742342423 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0025070 con 0x5618b161b820 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17d27fc640 1 -- 172.21.15.150:0/742342423 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e000fe30 con 0x5618b161b820 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.792+0000 7f17d27fc640 1 -- 172.21.15.150:0/742342423 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e002a750 con 0x5618b161b820 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- 172.21.15.150:0/742342423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 msgr2=0x5618b1617ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f17e00131d0 tx=0x7f17e0004ab0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.805 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- 172.21.15.150:0/742342423 shutdown_connections 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b1617ea0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b1630d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 --2- 172.21.15.150:0/742342423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b16272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- 172.21.15.150:0/742342423 >> 172.21.15.150:0/742342423 conn(0x5618b160ca50 msgr2=0x5618b160e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- 172.21.15.150:0/742342423 shutdown_connections 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- 172.21.15.150:0/742342423 wait complete. 2024-03-18T16:04:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 -- start start 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.793+0000 7f17f1dee740 1 Processor -- start 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1631060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b162e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81aba00 con 0x5618b161b820 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81ace70 con 0x5618b1612a50 2024-03-18T16:04:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81ac0d0 con 0x5618b1612130 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17f1dee740 10 client.0 _ll_register_callbacks cb 0x7f17e00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17ed6bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b162e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:45332/0 (socket says 172.21.15.150:45332) 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 -- 172.21.15.150:0/2526138092 learned_addr learned my addr 172.21.15.150:0/2526138092 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:04:55.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1631060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 -- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 msgr2=0x5618b1631060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1631060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 -- 172.21.15.150:0/2526138092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 msgr2=0x5618b162e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b162e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 -- 172.21.15.150:0/2526138092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b162f1a0 con 0x5618b1612a50 2024-03-18T16:04:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1631060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17ed6bf640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b162e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17e7fff640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009340 tx=0x7f17d4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb8686de0be963b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.794+0000 7f17c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d402c8b0 con 0x5618b1612a50 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8440+0+0 (secure 0 0 0) 0x7f17d402ca50 con 0x5618b1612a50 2024-03-18T16:04:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e000fe30 con 0x5618b1612a50 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17d4058020 con 0x5618b1612a50 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 10 client.25583 fetch_fsmap learned FSMap version 29 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618b16919c0 con 0x5618b1612a50 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e000fe30 con 0x5618b1612a50 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f17d404b3c0 con 0x5618b1612a50 2024-03-18T16:04:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 10 client.25583 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 20 client.25583 populate_metadata read hostname 'smithi150' 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b1692140 0x5618b1694560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 4 client.25583 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.795+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5618b157e2d0 con 0x5618b1692140 2024-03-18T16:04:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.796+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b1692140 0x5618b1694560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.796+0000 7f17ecebe640 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b1692140 0x5618b1694560 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c7038350535b9eb server_cookie=9936d8c35ac99f1 in_seq=0 out_seq=0 2024-03-18T16:04:55.813 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.796+0000 7f17c7fff640 10 client.25583 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:55.815 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.804+0000 7f17c7fff640 1 -- 172.21.15.150:0/2526138092 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5618b157e2d0 con 0x5618b1692140 2024-03-18T16:04:55.815 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.804+0000 7f17c7fff640 10 client.25583 handle_command_reply: tid=0 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.804+0000 7f17f1dee740 1 client.25583 shutdown 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 msgr2=0x5618b162e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009340 tx=0x7f17d4005be0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 shutdown_connections 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5618b1692140 0x5618b1694560 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618b161b820 0x5618b162e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17c7fff640 0 client.25583 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:04:55.816 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5618b1612a50 0x5618b162e3f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 --2- 172.21.15.150:0/2526138092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618b1612130 0x5618b1631060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 >> 172.21.15.150:0/2526138092 conn(0x5618b160ca50 msgr2=0x5618b1618ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 shutdown_connections 2024-03-18T16:04:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 1 -- 172.21.15.150:0/2526138092 wait complete. 2024-03-18T16:04:55.817 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17f1dee740 20 client.25583 trim_cache size 0 max 16384 2024-03-18T16:04:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:04:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:04:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:04:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:04:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 msgr2=0x7f17d80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 0x7f17d80a4ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x5618b160de70 tx=0x7f17e0009b00 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 msgr2=0x7f17e8198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f17dc00e9f0 tx=0x7f17dc00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 shutdown_connections 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17d80a2a10 0x7f17d80a4ed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8105680 0x7f17e819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17e81012f0 0x7f17e8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 --2- 172.21.15.150:0/2163019713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17e8100930 0x7f17e8198140 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.805+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 >> 172.21.15.150:0/2163019713 conn(0x7f17e8077530 msgr2=0x7f17e80ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.806+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 shutdown_connections 2024-03-18T16:04:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:04:55.806+0000 7f17ef122640 1 -- 172.21.15.150:0/2163019713 wait complete. 2024-03-18T16:04:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:55 smithi191 ceph-mon[36784]: pgmap v148: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:55 smithi156 ceph-mon[35919]: pgmap v148: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:55 smithi150 ceph-mon[31946]: pgmap v148: 97 pgs: 97 active+clean; 251 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 217 op/s 2024-03-18T16:04:56.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 458262 11.69 MB/sec execute 101 sec latency 15.045 ms 2024-03-18T16:04:57.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 459596 11.80 MB/sec execute 102 sec latency 63.137 ms 2024-03-18T16:04:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:57 smithi191 ceph-mon[36784]: pgmap v149: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 322 op/s 2024-03-18T16:04:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:57 smithi191 ceph-mon[36784]: evicting unresponsive client smithi150:x (15312), after 300.543 seconds 2024-03-18T16:04:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:57 smithi191 ceph-mon[36784]: Evicting (and blocklisting) client session 15312 (172.21.15.150:0/3130851108) 2024-03-18T16:04:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:57 smithi191 ceph-mon[36784]: from='mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183]' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:57 smithi191 ceph-mon[36784]: from='mds.? ' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:57 smithi156 ceph-mon[35919]: pgmap v149: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 322 op/s 2024-03-18T16:04:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:57 smithi156 ceph-mon[35919]: evicting unresponsive client smithi150:x (15312), after 300.543 seconds 2024-03-18T16:04:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:57 smithi156 ceph-mon[35919]: Evicting (and blocklisting) client session 15312 (172.21.15.150:0/3130851108) 2024-03-18T16:04:58.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:57 smithi156 ceph-mon[35919]: from='mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183]' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:58.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:57 smithi156 ceph-mon[35919]: from='mds.? ' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:57 smithi150 ceph-mon[31946]: pgmap v149: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 322 op/s 2024-03-18T16:04:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:57 smithi150 ceph-mon[31946]: evicting unresponsive client smithi150:x (15312), after 300.543 seconds 2024-03-18T16:04:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:57 smithi150 ceph-mon[31946]: Evicting (and blocklisting) client session 15312 (172.21.15.150:0/3130851108) 2024-03-18T16:04:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:57 smithi150 ceph-mon[31946]: from='mds.? [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183]' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:57 smithi150 ceph-mon[31946]: from='mds.? ' entity='mds.i' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]: dispatch 2024-03-18T16:04:58.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 462288 11.85 MB/sec execute 103 sec latency 11.686 ms 2024-03-18T16:04:58.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:58 smithi191 ceph-mon[36784]: from='mds.? ' entity='mds.i' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]': finished 2024-03-18T16:04:58.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:58 smithi191 ceph-mon[36784]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:04:59.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:58 smithi156 ceph-mon[35919]: from='mds.? ' entity='mds.i' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]': finished 2024-03-18T16:04:59.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:58 smithi156 ceph-mon[35919]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:04:59.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:58 smithi150 ceph-mon[31946]: from='mds.? ' entity='mds.i' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/3130851108"}]': finished 2024-03-18T16:04:59.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:58 smithi150 ceph-mon[31946]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:04:59.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 464620 11.84 MB/sec execute 104 sec latency 30.921 ms 2024-03-18T16:04:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:04:59 smithi191 ceph-mon[36784]: pgmap v151: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-18T16:05:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:04:59 smithi156 ceph-mon[35919]: pgmap v151: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-18T16:05:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:04:59 smithi150 ceph-mon[31946]: pgmap v151: 97 pgs: 97 active+clean; 257 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-18T16:05:00.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 466435 11.82 MB/sec execute 105 sec latency 45.729 ms 2024-03-18T16:05:00.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:00 smithi191 ceph-mon[36784]: pgmap v152: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:01.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:00 smithi156 ceph-mon[35919]: pgmap v152: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:01.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:00 smithi150 ceph-mon[31946]: pgmap v152: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:01.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 468784 11.83 MB/sec execute 106 sec latency 22.535 ms 2024-03-18T16:05:02.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 470156 11.78 MB/sec execute 107 sec latency 84.981 ms 2024-03-18T16:05:03.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 472854 11.83 MB/sec execute 108 sec latency 12.001 ms 2024-03-18T16:05:03.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:03 smithi191 ceph-mon[36784]: pgmap v153: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:03 smithi156 ceph-mon[35919]: pgmap v153: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:03 smithi150 ceph-mon[31946]: pgmap v153: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:04.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 475254 11.82 MB/sec execute 109 sec latency 24.936 ms 2024-03-18T16:05:05.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 477180 11.81 MB/sec execute 110 sec latency 13.992 ms 2024-03-18T16:05:05.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:05 smithi191 ceph-mon[36784]: pgmap v154: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:05 smithi156 ceph-mon[35919]: pgmap v154: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:05 smithi150 ceph-mon[31946]: pgmap v154: 97 pgs: 97 active+clean; 264 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 287 op/s 2024-03-18T16:05:06.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 479981 11.85 MB/sec execute 111 sec latency 11.729 ms 2024-03-18T16:05:07.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 481970 11.81 MB/sec execute 112 sec latency 67.228 ms 2024-03-18T16:05:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:07 smithi191 ceph-mon[36784]: pgmap v155: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 293 op/s 2024-03-18T16:05:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:07 smithi156 ceph-mon[35919]: pgmap v155: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 293 op/s 2024-03-18T16:05:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:07 smithi150 ceph-mon[31946]: pgmap v155: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 293 op/s 2024-03-18T16:05:08.323 INFO:tasks.workunit.client.0.smithi150.stdout: 1 483838 11.82 MB/sec execute 113 sec latency 11.769 ms 2024-03-18T16:05:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 -- 172.21.15.150:0/3080770525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 msgr2=0x7fc6240692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/3080770525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc6240692c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc614009930 tx=0x7fc61402bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 -- 172.21.15.150:0/3080770525 shutdown_connections 2024-03-18T16:05:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/3080770525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 0x7fc62410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/3080770525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc6240692c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/3080770525 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 0x7fc624107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 -- 172.21.15.150:0/3080770525 >> 172.21.15.150:0/3080770525 conn(0x7fc624077530 msgr2=0x7fc624077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 -- 172.21.15.150:0/3080770525 shutdown_connections 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.893+0000 7fc62bdf4640 1 -- 172.21.15.150:0/3080770525 wait complete. 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 Processor -- start 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 -- start start 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 0x7fc624198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 0x7fc62419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc624111ba0 con 0x7fc624068e60 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc624111a20 con 0x7fc624069800 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62bdf4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc624111d20 con 0x7fc6241078b0 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50858/0 (socket says 172.21.15.150:50858) 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 -- 172.21.15.150:0/2424864200 learned_addr learned my addr 172.21.15.150:0/2424864200 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62938f640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 0x7fc624198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc62a391640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 0x7fc62419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 -- 172.21.15.150:0/2424864200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 msgr2=0x7fc62419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 0x7fc62419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 -- 172.21.15.150:0/2424864200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 msgr2=0x7fc624198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 0x7fc624198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.894+0000 7fc629b90640 1 -- 172.21.15.150:0/2424864200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62419cfc0 con 0x7fc624068e60 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc629b90640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc61affd640 1 -- 172.21.15.150:0/2424864200 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62000cd80 con 0x7fc624068e60 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62419d2b0 con 0x7fc624068e60 2024-03-18T16:05:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc61affd640 1 -- 172.21.15.150:0/2424864200 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc620005500 con 0x7fc624068e60 2024-03-18T16:05:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc61affd640 1 -- 172.21.15.150:0/2424864200 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620010690 con 0x7fc624068e60 2024-03-18T16:05:08.901 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.895+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6241a4b40 con 0x7fc624068e60 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.897+0000 7fc61affd640 1 -- 172.21.15.150:0/2424864200 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fc6200046c0 con 0x7fc624068e60 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.897+0000 7fc61affd640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 0x7fc6100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 Processor -- start 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62938f640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 0x7fc6100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc61affd640 1 -- 172.21.15.150:0/2424864200 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fc6200cf230 con 0x7fc624068e60 2024-03-18T16:05:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 -- start start 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 0x55581b9d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9dbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55581b934a40 con 0x55581b9d0130 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55581b8dd920 con 0x55581b9d0a50 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55581b930ae0 con 0x55581b9d9820 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50872/0 (socket says 172.21.15.150:50872) 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 -- 172.21.15.150:0/2256311622 learned_addr learned my addr 172.21.15.150:0/2256311622 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62a391640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9dbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62938f640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 0x7fc6100a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc614005ab0 tx=0x7fc614000bc0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62938f640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 0x55581b9d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 -- 172.21.15.150:0/2256311622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 msgr2=0x55581b9dbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9dbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 -- 172.21.15.150:0/2256311622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 msgr2=0x55581b9d92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 0x55581b9d92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc629b90640 1 -- 172.21.15.150:0/2256311622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55581b8f5460 con 0x55581b9d0130 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.898+0000 7fc62938f640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 0x55581b9d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.899+0000 7fc629b90640 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc620009380 tx=0x7fc62009b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff19be75ea0f706 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.899+0000 7fc60ffff640 1 -- 172.21.15.150:0/2256311622 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62009fa00 con 0x55581b9d0130 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.899+0000 7fc60ffff640 1 -- 172.21.15.150:0/2256311622 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6200a4050 con 0x55581b9d0130 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.899+0000 7fc60ffff640 1 -- 172.21.15.150:0/2256311622 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62009ce20 con 0x55581b9d0130 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- 172.21.15.150:0/2256311622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 msgr2=0x55581b9d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc620009380 tx=0x7fc62009b210 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- 172.21.15.150:0/2256311622 shutdown_connections 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9dbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0a50 0x55581b9d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/2256311622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0130 0x55581b9d0510 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- 172.21.15.150:0/2256311622 >> 172.21.15.150:0/2256311622 conn(0x55581b9caa50 msgr2=0x55581b9cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- 172.21.15.150:0/2256311622 shutdown_connections 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- 172.21.15.150:0/2256311622 wait complete. 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 -- start start 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.900+0000 7fc62e9dd740 1 Processor -- start 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55581b9dc450 con 0x55581b9d9820 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55581b8f0d70 con 0x55581b9d0130 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55581b9dc5d0 con 0x55581b9d0a50 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62e9dd740 10 client.0 _ll_register_callbacks cb 0x55581b648a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50880/0 (socket says 172.21.15.150:50880) 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 -- 172.21.15.150:0/1722388142 learned_addr learned my addr 172.21.15.150:0/1722388142 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62938f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc629b90640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 -- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 msgr2=0x55581b9d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 -- 172.21.15.150:0/1722388142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 msgr2=0x55581b9d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 -- 172.21.15.150:0/1722388142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55581b649210 con 0x55581b9d9820 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc62a391640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc61c011d10 tx=0x7fc61c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383f5d532f1371a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.901+0000 7fc60d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c00fe60 con 0x55581b9d9820 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fc61c003970 con 0x55581b9d9820 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6200a4050 con 0x55581b9d9820 2024-03-18T16:05:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc61c003b70 con 0x55581b9d9820 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 10 client.15828 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55581b715620 con 0x55581b9d9820 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6200a4050 con 0x55581b9d9820 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc61c022070 con 0x55581b9d9820 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 10 client.15828 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 20 client.15828 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581b9e9ab0 0x55581b9ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 4 client.15828 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55581b941f80 con 0x55581b9e9ab0 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.902+0000 7fc629b90640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581b9e9ab0 0x55581b9ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.903+0000 7fc629b90640 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581b9e9ab0 0x55581b9ebec0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=320f5866d2acf96c server_cookie=323e11f54922884d in_seq=0 out_seq=0 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.903+0000 7fc60d7fa640 10 client.15828 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1722388142 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55581b941f80 con 0x55581b9e9ab0 2024-03-18T16:05:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc60d7fa640 10 client.15828 handle_command_reply: tid=0 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 client.15828 shutdown 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 msgr2=0x55581b9d7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc61c011d10 tx=0x7fc61c00f790 comp rx=0 tx=0).stop 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 shutdown_connections 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581b9e9ab0 0x55581b9ebec0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9d7330 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1722388142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc60d7fa640 0 client.15828 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 >> 172.21.15.150:0/1722388142 conn(0x55581b9caa50 msgr2=0x55581b9cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 shutdown_connections 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1722388142 wait complete. 2024-03-18T16:05:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.904+0000 7fc62e9dd740 20 client.15828 trim_cache size 0 max 16384 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 Processor -- start 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 -- start start 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9e1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241afcb0 con 0x55581b9d0a50 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241ad580 con 0x55581b9d0130 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241ab3f0 con 0x55581b9d9820 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50896/0 (socket says 172.21.15.150:50896) 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 -- 172.21.15.150:0/960317567 learned_addr learned my addr 172.21.15.150:0/960317567 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc629b90640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62a391640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 -- 172.21.15.150:0/960317567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 msgr2=0x55581b9e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9e1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 -- 172.21.15.150:0/960317567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 msgr2=0x55581b9e51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 -- 172.21.15.150:0/960317567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55581b9c4fc0 con 0x55581b9d0a50 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc629b90640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc62938f640 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc614036040 tx=0x7fc61403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba991b2a496183 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.919+0000 7fc60f7fe640 1 -- 172.21.15.150:0/960317567 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61402ccb0 con 0x55581b9d0a50 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc60f7fe640 1 -- 172.21.15.150:0/960317567 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc614051070 con 0x55581b9d0a50 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc60f7fe640 1 -- 172.21.15.150:0/960317567 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6140035f0 con 0x55581b9d0a50 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- 172.21.15.150:0/960317567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 msgr2=0x55581b9f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc614036040 tx=0x7fc61403f000 comp rx=0 tx=0).stop 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- 172.21.15.150:0/960317567 shutdown_connections 2024-03-18T16:05:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d9820 0x55581b9e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d0a50 0x55581b9f17d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/960317567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- 172.21.15.150:0/960317567 >> 172.21.15.150:0/960317567 conn(0x55581b9caa50 msgr2=0x55581b9c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- 172.21.15.150:0/960317567 shutdown_connections 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- 172.21.15.150:0/960317567 wait complete. 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.920+0000 7fc62e9dd740 1 -- start start 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 Processor -- start 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9de070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241aba40 con 0x55581b9d9820 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241aceb0 con 0x55581b9d0130 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241ac110 con 0x55581b9d0a50 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62e9dd740 10 client.0 _ll_register_callbacks cb 0x7fc61c009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62a391640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9de070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:57850/0 (socket says 172.21.15.150:57850) 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 -- 172.21.15.150:0/1717377514 learned_addr learned my addr 172.21.15.150:0/1717377514 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc629b90640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 -- 172.21.15.150:0/1717377514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 msgr2=0x55581b9e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 -- 172.21.15.150:0/1717377514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 msgr2=0x55581b9de070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9de070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62938f640 1 -- 172.21.15.150:0/1717377514 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55581b9de5b0 con 0x55581b9d0a50 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.921+0000 7fc62a391640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9de070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62938f640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc614041040 tx=0x7fc614005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc38676b4563f21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614005310 con 0x55581b9d0a50 2024-03-18T16:05:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc614051070 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fc6140054b0 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc6140046c0 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 10 client.25657 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55581ba50b10 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc614051070 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fc614003b70 con 0x55581b9d0a50 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 10 client.25657 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 20 client.25657 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581ba51150 0x55581ba53570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 4 client.25657 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55581b93c2d0 con 0x55581ba51150 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.922+0000 7fc629b90640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581ba51150 0x55581ba53570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.923+0000 7fc60d7fa640 10 client.25657 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:08.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.923+0000 7fc629b90640 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581ba51150 0x55581ba53570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7f8a636caf0a8bd server_cookie=77dafdfe35486a41 in_seq=0 out_seq=0 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc60d7fa640 1 -- 172.21.15.150:0/1717377514 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55581b93c2d0 con 0x55581ba51150 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc60d7fa640 10 client.25657 handle_command_reply: tid=0 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 client.25657 shutdown 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 msgr2=0x55581b9ddb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc614041040 tx=0x7fc614005020 comp rx=0 tx=0).stop 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 shutdown_connections 2024-03-18T16:05:08.936 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55581ba51150 0x55581ba53570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55581b9d9820 0x55581b9de070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc60d7fa640 0 client.25657 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55581b9d0a50 0x55581b9ddb30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 --2- 172.21.15.150:0/1717377514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55581b9d0130 0x55581b9e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 >> 172.21.15.150:0/1717377514 conn(0x55581b9caa50 msgr2=0x55581b9cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 shutdown_connections 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 1 -- 172.21.15.150:0/1717377514 wait complete. 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.933+0000 7fc62e9dd740 20 client.25657 trim_cache size 0 max 16384 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:05:08.937 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:05:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:05:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 msgr2=0x7fc6100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 0x7fc6100a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc614005ab0 tx=0x7fc614000bc0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 msgr2=0x7fc624198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000edf0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 shutdown_connections 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fc6100a2a10 0x7fc6100a4ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc6241078b0 0x7fc62419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc624069800 0x7fc624198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 --2- 172.21.15.150:0/2424864200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc624068e60 0x7fc624198020 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 >> 172.21.15.150:0/2424864200 conn(0x7fc624077530 msgr2=0x7fc6240fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 shutdown_connections 2024-03-18T16:05:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:08.934+0000 7fc62bdf4640 1 -- 172.21.15.150:0/2424864200 wait complete. 2024-03-18T16:05:09.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 486565 11.84 MB/sec execute 114 sec latency 13.724 ms 2024-03-18T16:05:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:09 smithi191 ceph-mon[36784]: pgmap v156: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T16:05:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:09 smithi156 ceph-mon[35919]: pgmap v156: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T16:05:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:09 smithi150 ceph-mon[31946]: pgmap v156: 97 pgs: 97 active+clean; 271 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 270 op/s 2024-03-18T16:05:10.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 488456 11.82 MB/sec execute 115 sec latency 18.047 ms 2024-03-18T16:05:11.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 490773 11.84 MB/sec execute 116 sec latency 15.776 ms 2024-03-18T16:05:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:11 smithi191 ceph-mon[36784]: pgmap v157: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 351 op/s 2024-03-18T16:05:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:11 smithi156 ceph-mon[35919]: pgmap v157: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 351 op/s 2024-03-18T16:05:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:11 smithi150 ceph-mon[31946]: pgmap v157: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 351 op/s 2024-03-18T16:05:12.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 492787 11.81 MB/sec execute 117 sec latency 66.019 ms 2024-03-18T16:05:13.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 494652 11.81 MB/sec execute 118 sec latency 13.191 ms 2024-03-18T16:05:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:13 smithi191 ceph-mon[36784]: pgmap v158: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:13 smithi156 ceph-mon[35919]: pgmap v158: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:13 smithi150 ceph-mon[31946]: pgmap v158: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:14.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 497378 11.85 MB/sec execute 119 sec latency 65.389 ms 2024-03-18T16:05:14.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:14 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:15.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:14 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:15.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:14 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:15.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 499092 11.81 MB/sec execute 120 sec latency 14.986 ms 2024-03-18T16:05:15.920 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:15 smithi156 ceph-mon[35919]: pgmap v159: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:15.920 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:15 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:15.920 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:15 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:15.920 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:15 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:05:15.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:15 smithi191 ceph-mon[36784]: pgmap v159: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:15 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:15 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:15 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:05:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:15 smithi150 ceph-mon[31946]: pgmap v159: 97 pgs: 97 active+clean; 278 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-03-18T16:05:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:15 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:15 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:16.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:15 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:05:16.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 501495 11.83 MB/sec execute 121 sec latency 10.999 ms 2024-03-18T16:05:17.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 503724 11.82 MB/sec execute 122 sec latency 95.188 ms 2024-03-18T16:05:17.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:17 smithi191 ceph-mon[36784]: pgmap v160: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 318 op/s 2024-03-18T16:05:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:17 smithi156 ceph-mon[35919]: pgmap v160: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 318 op/s 2024-03-18T16:05:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:17 smithi150 ceph-mon[31946]: pgmap v160: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 318 op/s 2024-03-18T16:05:18.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 505627 11.81 MB/sec execute 123 sec latency 18.046 ms 2024-03-18T16:05:19.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 508327 11.85 MB/sec execute 124 sec latency 13.067 ms 2024-03-18T16:05:19.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:19 smithi191 ceph-mon[36784]: pgmap v161: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-18T16:05:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:19 smithi156 ceph-mon[35919]: pgmap v161: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-18T16:05:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:19 smithi150 ceph-mon[31946]: pgmap v161: 97 pgs: 97 active+clean; 284 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-18T16:05:20.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 510642 11.82 MB/sec execute 125 sec latency 18.775 ms 2024-03-18T16:05:21.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 512458 11.82 MB/sec execute 126 sec latency 24.132 ms 2024-03-18T16:05:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:21 smithi191 ceph-mon[36784]: pgmap v162: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 311 op/s 2024-03-18T16:05:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:21 smithi156 ceph-mon[35919]: pgmap v162: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 311 op/s 2024-03-18T16:05:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- 172.21.15.150:0/3631870915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 msgr2=0x7f2180105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3631870915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f2180105a60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2170009900 tx=0x7f217002bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- 172.21.15.150:0/3631870915 shutdown_connections 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3631870915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21801012f0 0x7f218010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3631870915 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2180100930 0x7f2180100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3631870915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f2180105a60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- 172.21.15.150:0/3631870915 >> 172.21.15.150:0/3631870915 conn(0x7f2180077530 msgr2=0x7f2180077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- 172.21.15.150:0/3631870915 shutdown_connections 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- 172.21.15.150:0/3631870915 wait complete. 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 Processor -- start 2024-03-18T16:05:22.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.028+0000 7f2185ee8640 1 -- start start 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2180100930 0x7f2180196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21801012f0 0x7f2180197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2180110ba0 con 0x7f2180105680 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2180110a20 con 0x7f2180100930 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2180110d20 con 0x7f21801012f0 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46592/0 (socket says 172.21.15.150:46592) 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 -- 172.21.15.150:0/3886215226 learned_addr learned my addr 172.21.15.150:0/3886215226 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217f7fe640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2180100930 0x7f2180196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2177fff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21801012f0 0x7f2180197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 -- 172.21.15.150:0/3886215226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21801012f0 msgr2=0x7f2180197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21801012f0 0x7f2180197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 -- 172.21.15.150:0/3886215226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2180100930 msgr2=0x7f2180196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2180100930 0x7f2180196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 -- 172.21.15.150:0/3886215226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218019bf60 con 0x7f2180105680 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217ffff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f217800ea60 tx=0x7f217800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217d7fa640 1 -- 172.21.15.150:0/3886215226 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217800ce40 con 0x7f2180105680 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217d7fa640 1 -- 172.21.15.150:0/3886215226 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2178005020 con 0x7f2180105680 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218019c250 con 0x7f2180105680 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f217d7fa640 1 -- 172.21.15.150:0/3886215226 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178010690 con 0x7f2180105680 2024-03-18T16:05:22.034 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.029+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21801a3ae0 con 0x7f2180105680 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.031+0000 7f217d7fa640 1 -- 172.21.15.150:0/3886215226 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f2178005520 con 0x7f2180105680 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 Processor -- start 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 -- start start 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 0x562f3707b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f3707ba30 0x562f3707de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562f36fd4a40 con 0x562f37072340 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562f36f7d920 con 0x562f3707ba30 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2188baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562f36fd0ae0 con 0x562f37072c60 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2177fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 0x562f3707b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217d7fa640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 0x7f216c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2177fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46608/0 (socket says 172.21.15.150:46608) 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2177fff640 1 -- 172.21.15.150:0/2842572495 learned_addr learned my addr 172.21.15.150:0/2842572495 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2177fff640 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f3707ba30 0x562f3707de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217ffff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 0x7f216c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 -- 172.21.15.150:0/2842572495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f3707ba30 msgr2=0x562f3707de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f3707ba30 0x562f3707de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 -- 172.21.15.150:0/2842572495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 msgr2=0x562f37072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f217f7fe640 1 -- 172.21.15.150:0/2842572495 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f36f95460 con 0x562f37072c60 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.032+0000 7f2177fff640 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f217d7fa640 1 -- 172.21.15.150:0/3886215226 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f21780cf310 con 0x7f2180105680 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f217f7fe640 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 0x562f3707b4f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2170000c00 tx=0x7f2170005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b9f46765d35c65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f217ffff640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 0x7f216c0a4ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2178097df0 tx=0x7f217809e040 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f2175ffb640 1 -- 172.21.15.150:0/2842572495 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170042070 con 0x562f37072c60 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f2175ffb640 1 -- 172.21.15.150:0/2842572495 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f217002c8b0 con 0x562f37072c60 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.033+0000 7f2175ffb640 1 -- 172.21.15.150:0/2842572495 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170004c20 con 0x562f37072c60 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- 172.21.15.150:0/2842572495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 msgr2=0x562f3707b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 0x562f3707b4f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2170000c00 tx=0x7f2170005be0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- 172.21.15.150:0/2842572495 shutdown_connections 2024-03-18T16:05:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f3707ba30 0x562f3707de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f37072c60 0x562f3707b4f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 --2- 172.21.15.150:0/2842572495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- 172.21.15.150:0/2842572495 >> 172.21.15.150:0/2842572495 conn(0x562f3706cc60 msgr2=0x562f3706d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- 172.21.15.150:0/2842572495 shutdown_connections 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- 172.21.15.150:0/2842572495 wait complete. 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 -- start start 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.034+0000 7f2188baa740 1 Processor -- start 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072340 0x562f37078a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f370794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562f3707e600 con 0x562f37072c60 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562f36f90d70 con 0x562f37072340 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562f3707e780 con 0x562f3707ba30 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2188baa740 10 client.0 _ll_register_callbacks cb 0x562f36ce8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46620/0 (socket says 172.21.15.150:46620) 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 -- 172.21.15.150:0/840012401 learned_addr learned my addr 172.21.15.150:0/840012401 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2177fff640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f370794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f2177fff640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072340 0x562f37078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 -- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 msgr2=0x562f370794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f370794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 -- 172.21.15.150:0/840012401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072340 msgr2=0x562f37078a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072340 0x562f37078a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 -- 172.21.15.150:0/840012401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f36ce9210 con 0x562f37072c60 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f217f7fe640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2170036040 tx=0x7f2170004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b774247c76410adb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f215effd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.035+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170042040 con 0x562f37072c60 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f2170005000 con 0x562f37072c60 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f217002c8b0 con 0x562f37072c60 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2170005200 con 0x562f37072c60 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f36db5620 con 0x562f37072c60 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f217002c8b0 con 0x562f37072c60 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2170047c40 con 0x562f37072c60 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 20 client.15846 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f3708ba00 0x562f3708ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 4 client.15846 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f36fe1f80 con 0x562f3708ba00 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.036+0000 7f2177fff640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f3708ba00 0x562f3708ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.037+0000 7f2177fff640 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f3708ba00 0x562f3708ddf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b310f6e9f143e282 server_cookie=c8913644c8c21b85 in_seq=0 out_seq=0 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.037+0000 7f215effd640 10 client.15846 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f215effd640 1 -- 172.21.15.150:0/840012401 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562f36fe1f80 con 0x562f3708ba00 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f215effd640 10 client.15846 handle_command_reply: tid=0 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 client.15846 shutdown 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 msgr2=0x562f37078fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2170036040 tx=0x7f2170004bd0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 shutdown_connections 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f3708ba00 0x562f3708ddf0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f370794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072c60 0x562f37078fb0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f215effd640 0 client.15846 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 --2- 172.21.15.150:0/840012401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072340 0x562f37078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 >> 172.21.15.150:0/840012401 conn(0x562f3706cc60 msgr2=0x562f37069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 shutdown_connections 2024-03-18T16:05:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 1 -- 172.21.15.150:0/840012401 wait complete. 2024-03-18T16:05:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.038+0000 7f2188baa740 20 client.15846 trim_cache size 0 max 16384 2024-03-18T16:05:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.066+0000 7f2188baa740 1 Processor -- start 2024-03-18T16:05:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 -- start start 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f3707b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370873a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37093530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21801ad8b0 con 0x562f37072340 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21801ab180 con 0x562f37072c60 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.067+0000 7f2188baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a8ff0 con 0x562f3707ba30 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.068+0000 7f2177fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37093530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.068+0000 7f2177fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f3707b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.068+0000 7f2177fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f3707b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46626/0 (socket says 172.21.15.150:46626) 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.068+0000 7f2177fff640 1 -- 172.21.15.150:0/846576307 learned_addr learned my addr 172.21.15.150:0/846576307 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.068+0000 7f217f7fe640 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 -- 172.21.15.150:0/846576307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 msgr2=0x562f370873a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370873a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 -- 172.21.15.150:0/846576307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 msgr2=0x562f3707b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f3707b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 -- 172.21.15.150:0/846576307 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f37071c90 con 0x562f3707ba30 2024-03-18T16:05:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.070+0000 7f2177fff640 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37093530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2168000c00 tx=0x7f2168009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c53ec83024abbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.071+0000 7f21757fa640 1 -- 172.21.15.150:0/846576307 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21680139c0 con 0x562f3707ba30 2024-03-18T16:05:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.071+0000 7f21757fa640 1 -- 172.21.15.150:0/846576307 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168026810 con 0x562f3707ba30 2024-03-18T16:05:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.071+0000 7f21757fa640 1 -- 172.21.15.150:0/846576307 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168003b20 con 0x562f3707ba30 2024-03-18T16:05:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 -- 172.21.15.150:0/846576307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 msgr2=0x562f37093530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37093530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2168000c00 tx=0x7f2168009780 comp rx=0 tx=0).stop 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 -- 172.21.15.150:0/846576307 shutdown_connections 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37093530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 --2- 172.21.15.150:0/846576307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f3707b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 -- 172.21.15.150:0/846576307 >> 172.21.15.150:0/846576307 conn(0x562f3706cc60 msgr2=0x562f3706bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 -- 172.21.15.150:0/846576307 shutdown_connections 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.072+0000 7f2188baa740 1 -- 172.21.15.150:0/846576307 wait complete. 2024-03-18T16:05:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 -- start start 2024-03-18T16:05:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 Processor -- start 2024-03-18T16:05:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37083cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370753c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37075940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a9640 con 0x562f37072340 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21801aaab0 con 0x562f37072c60 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a9d10 con 0x562f3707ba30 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.073+0000 7f2188baa740 10 client.0 _ll_register_callbacks cb 0x7f2168009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.074+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.074+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370753c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:49126/0 (socket says 172.21.15.150:49126) 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.074+0000 7f217f7fe640 1 -- 172.21.15.150:0/2986177321 learned_addr learned my addr 172.21.15.150:0/2986177321 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.074+0000 7f2177fff640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.074+0000 7f2177fff640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37083cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 -- 172.21.15.150:0/2986177321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 msgr2=0x562f370753c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370753c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 -- 172.21.15.150:0/2986177321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 msgr2=0x562f37083cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37083cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 -- 172.21.15.150:0/2986177321 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f37076170 con 0x562f3707ba30 2024-03-18T16:05:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.075+0000 7f2177fff640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37075940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f216800a010 tx=0x7f2168003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867bb8362f093719 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.076+0000 7f215effd640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.076+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168026490 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.076+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f21680038a0 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.076+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2168026810 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.076+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2168026630 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.077+0000 7f2188baa740 10 client.25675 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.077+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562f37100c20 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.077+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168026810 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.077+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2168023070 con 0x562f3707ba30 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 10 client.25675 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 10 client.25675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 20 client.25675 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f37101280 0x562f37103670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 4 client.25675 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562f36fdc2d0 con 0x562f37101280 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.078+0000 7f217f7fe640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f37101280 0x562f37103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.079+0000 7f217f7fe640 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f37101280 0x562f37103670 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b137db80d333a4 server_cookie=9a79d781008383e in_seq=0 out_seq=0 2024-03-18T16:05:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.079+0000 7f215effd640 10 client.25675 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.088+0000 7f215effd640 1 -- 172.21.15.150:0/2986177321 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562f36fdc2d0 con 0x562f37101280 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.088+0000 7f215effd640 10 client.25675 handle_command_reply: tid=0 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.088+0000 7f2188baa740 1 client.25675 shutdown 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 msgr2=0x562f37075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37075940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f216800a010 tx=0x7f2168003700 comp rx=0 tx=0).stop 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 shutdown_connections 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562f37101280 0x562f37103670 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562f3707ba30 0x562f37075940 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562f37072c60 0x562f370753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f215effd640 0 client.25675 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 --2- 172.21.15.150:0/2986177321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562f37072340 0x562f37083cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 >> 172.21.15.150:0/2986177321 conn(0x562f3706cc60 msgr2=0x562f3706b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 shutdown_connections 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 1 -- 172.21.15.150:0/2986177321 wait complete. 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2188baa740 20 client.25675 trim_cache size 0 max 16384 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:05:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:05:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 msgr2=0x7f216c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 0x7f216c0a4ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2178097df0 tx=0x7f217809e040 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 msgr2=0x7f218019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.089+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f217800ea60 tx=0x7f217800ef20 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 shutdown_connections 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f216c0a2a10 0x7f216c0a4ed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2180105680 0x7f218019b830 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21801012f0 0x7f2180197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 --2- 172.21.15.150:0/3886215226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2180100930 0x7f2180196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 >> 172.21.15.150:0/3886215226 conn(0x7f2180077530 msgr2=0x7f21800fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 shutdown_connections 2024-03-18T16:05:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:22.090+0000 7f2185ee8640 1 -- 172.21.15.150:0/3886215226 wait complete. 2024-03-18T16:05:22.095 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:21 smithi150 ceph-mon[31946]: pgmap v162: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 311 op/s 2024-03-18T16:05:22.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 514921 11.83 MB/sec execute 127 sec latency 89.647 ms 2024-03-18T16:05:23.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 516643 11.82 MB/sec execute 128 sec latency 19.547 ms 2024-03-18T16:05:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:23 smithi191 ceph-mon[36784]: pgmap v163: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:23 smithi156 ceph-mon[35919]: pgmap v163: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:23 smithi150 ceph-mon[31946]: pgmap v163: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:24.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 519119 11.83 MB/sec execute 129 sec latency 12.731 ms 2024-03-18T16:05:25.324 INFO:tasks.workunit.client.0.smithi150.stdout: 1 521846 11.84 MB/sec execute 130 sec latency 19.357 ms 2024-03-18T16:05:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:25 smithi191 ceph-mon[36784]: pgmap v164: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:25 smithi156 ceph-mon[35919]: pgmap v164: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:25 smithi150 ceph-mon[31946]: pgmap v164: 97 pgs: 97 active+clean; 292 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 205 op/s 2024-03-18T16:05:26.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 523665 11.84 MB/sec execute 131 sec latency 19.931 ms 2024-03-18T16:05:27.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 526025 11.85 MB/sec execute 132 sec latency 87.941 ms 2024-03-18T16:05:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:27 smithi191 ceph-mon[36784]: pgmap v165: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 313 op/s 2024-03-18T16:05:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:27 smithi156 ceph-mon[35919]: pgmap v165: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 313 op/s 2024-03-18T16:05:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:27 smithi150 ceph-mon[31946]: pgmap v165: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 313 op/s 2024-03-18T16:05:28.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 528153 11.82 MB/sec execute 133 sec latency 42.483 ms 2024-03-18T16:05:29.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 529983 11.82 MB/sec execute 134 sec latency 15.850 ms 2024-03-18T16:05:29.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:29 smithi191 ceph-mon[36784]: pgmap v166: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-18T16:05:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:29 smithi156 ceph-mon[35919]: pgmap v166: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-18T16:05:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:29 smithi150 ceph-mon[31946]: pgmap v166: 97 pgs: 97 active+clean; 299 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 211 op/s 2024-03-18T16:05:30.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 532709 11.84 MB/sec execute 135 sec latency 29.078 ms 2024-03-18T16:05:31.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 534356 11.82 MB/sec execute 136 sec latency 19.124 ms 2024-03-18T16:05:31.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:31 smithi191 ceph-mon[36784]: pgmap v167: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 316 op/s 2024-03-18T16:05:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:31 smithi156 ceph-mon[35919]: pgmap v167: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 316 op/s 2024-03-18T16:05:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:31 smithi150 ceph-mon[31946]: pgmap v167: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 316 op/s 2024-03-18T16:05:32.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 536726 11.84 MB/sec execute 137 sec latency 20.092 ms 2024-03-18T16:05:33.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 538079 11.79 MB/sec execute 138 sec latency 76.276 ms 2024-03-18T16:05:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:33 smithi191 ceph-mon[36784]: pgmap v168: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:33 smithi156 ceph-mon[35919]: pgmap v168: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:33 smithi150 ceph-mon[31946]: pgmap v168: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:34.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 540286 11.81 MB/sec execute 139 sec latency 15.059 ms 2024-03-18T16:05:35.193 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 -- 172.21.15.150:0/3595153653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100a10 msgr2=0x7ff694104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.193 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/3595153653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100a10 0x7ff694104a50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff680009900 tx=0x7ff68002bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:35.193 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 -- 172.21.15.150:0/3595153653 shutdown_connections 2024-03-18T16:05:35.193 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/3595153653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff694105130 0x7ff69410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/3595153653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100a10 0x7ff694104a50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/3595153653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694100060 0x7ff694100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.191+0000 7ff69aa6c640 1 -- 172.21.15.150:0/3595153653 >> 172.21.15.150:0/3595153653 conn(0x7ff694076c70 msgr2=0x7ff6940fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.192+0000 7ff69aa6c640 1 -- 172.21.15.150:0/3595153653 shutdown_connections 2024-03-18T16:05:35.194 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.192+0000 7ff69aa6c640 1 -- 172.21.15.150:0/3595153653 wait complete. 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.192+0000 7ff69aa6c640 1 Processor -- start 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.192+0000 7ff69aa6c640 1 -- start start 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.192+0000 7ff69aa6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694105130 0x7ff6941985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff69419c970 0x7ff69419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410f960 con 0x7ff694100060 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410f7e0 con 0x7ff694105130 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff69410fae0 con 0x7ff69419c970 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49744/0 (socket says 172.21.15.150:49744) 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 -- 172.21.15.150:0/533546229 learned_addr learned my addr 172.21.15.150:0/533546229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff699009640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff69419c970 0x7ff69419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff693fff640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694105130 0x7ff6941985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 -- 172.21.15.150:0/533546229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff69419c970 msgr2=0x7ff69419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff69419c970 0x7ff69419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 -- 172.21.15.150:0/533546229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694105130 msgr2=0x7ff6941985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694105130 0x7ff6941985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 -- 172.21.15.150:0/533546229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69419d460 con 0x7ff694100060 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff698808640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff68800ea20 tx=0x7ff68800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff691ffb640 1 -- 172.21.15.150:0/533546229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68800cd40 con 0x7ff694100060 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.193+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69419d750 con 0x7ff694100060 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.194+0000 7ff691ffb640 1 -- 172.21.15.150:0/533546229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6880168f0 con 0x7ff694100060 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.194+0000 7ff691ffb640 1 -- 172.21.15.150:0/533546229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68800b920 con 0x7ff694100060 2024-03-18T16:05:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.194+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6941a4f60 con 0x7ff694100060 2024-03-18T16:05:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.196+0000 7ff691ffb640 1 -- 172.21.15.150:0/533546229 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7ff688016aa0 con 0x7ff694100060 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 Processor -- start 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 -- start start 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff691ffb640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 0x7ff6840a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac3204afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 0x55ac320586f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff691ffb640 1 -- 172.21.15.150:0/533546229 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7ff6880cec20 con 0x7ff694100060 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff699009640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 0x7ff6840a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ac31fafa40 con 0x55ac32058c30 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ac31f58920 con 0x55ac3204b4f0 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff69d64c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ac31fabae0 con 0x55ac3204abd0 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff699009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff699009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49758/0 (socket says 172.21.15.150:49758) 2024-03-18T16:05:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff699009640 1 -- 172.21.15.150:0/563811664 learned_addr learned my addr 172.21.15.150:0/563811664 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac3204afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff693fff640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 0x55ac320586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 -- 172.21.15.150:0/563811664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 msgr2=0x55ac320586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 0x55ac320586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 -- 172.21.15.150:0/563811664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 msgr2=0x55ac3205b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff698808640 1 -- 172.21.15.150:0/563811664 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac31f70460 con 0x55ac3204abd0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff699009640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.197+0000 7ff693fff640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 0x55ac320586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff699009640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 0x7ff6840a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff68c013c70 tx=0x7ff68c009e90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff698808640 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac3204afb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6880cc390 tx=0x7ff68809ad70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e64f97254f06cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff67effd640 1 -- 172.21.15.150:0/563811664 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688099650 con 0x55ac3204abd0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff67effd640 1 -- 172.21.15.150:0/563811664 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6880cc610 con 0x55ac3204abd0 2024-03-18T16:05:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff67effd640 1 -- 172.21.15.150:0/563811664 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68809a7a0 con 0x55ac3204abd0 2024-03-18T16:05:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff69d64c740 1 -- 172.21.15.150:0/563811664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 msgr2=0x55ac3204afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.198+0000 7ff69d64c740 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac3204afb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6880cc390 tx=0x7ff68809ad70 comp rx=0 tx=0).stop 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- 172.21.15.150:0/563811664 shutdown_connections 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac3205b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204b4f0 0x55ac320586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- 172.21.15.150:0/563811664 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac3204afb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- 172.21.15.150:0/563811664 >> 172.21.15.150:0/563811664 conn(0x55ac32045a50 msgr2=0x55ac32045e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- 172.21.15.150:0/563811664 shutdown_connections 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- 172.21.15.150:0/563811664 wait complete. 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- start start 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 Processor -- start 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac32055c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 0x55ac320566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55ac3205b7a0 con 0x55ac3204b4f0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ac31f6bd70 con 0x55ac32058c30 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55ac3205b920 con 0x55ac3204abd0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff69d64c740 10 client.0 _ll_register_callbacks cb 0x55ac31cc3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.199+0000 7ff693fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49764/0 (socket says 172.21.15.150:49764) 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 -- 172.21.15.150:0/597929234 learned_addr learned my addr 172.21.15.150:0/597929234 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff699009640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 0x55ac320566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff698808640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac32055c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 -- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 msgr2=0x55ac32055c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac32055c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 -- 172.21.15.150:0/597929234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 msgr2=0x55ac320566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 0x55ac320566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 -- 172.21.15.150:0/597929234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac31cc4210 con 0x55ac3204b4f0 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff699009640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 0x55ac320566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff693fff640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff694073f60 tx=0x7ff680009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8a9a84ad231860 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff673fff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680004300 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7ff6800046c0 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6880cc610 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6800044a0 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff69d64c740 10 client.15864 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac31d90620 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.200+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6880cc610 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ff68004e070 con 0x55ac3204b4f0 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 10 client.15864 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 20 client.15864 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac32068e40 0x55ac3206b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 4 client.15864 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac31fbcf80 con 0x55ac32068e40 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff698808640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac32068e40 0x55ac3206b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff698808640 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac32068e40 0x55ac3206b250 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34e88aeeca512fa8 server_cookie=63e46657c0660a80 in_seq=0 out_seq=0 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.201+0000 7ff673fff640 10 client.15864 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff673fff640 1 -- 172.21.15.150:0/597929234 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac31fbcf80 con 0x55ac32068e40 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff673fff640 10 client.15864 handle_command_reply: tid=0 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 client.15864 shutdown 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 msgr2=0x55ac32056180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff694073f60 tx=0x7ff680009560 comp rx=0 tx=0).stop 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 shutdown_connections 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac32068e40 0x55ac3206b250 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac32058c30 0x55ac320566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32056180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff673fff640 0 client.15864 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 --2- 172.21.15.150:0/597929234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204abd0 0x55ac32055c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 >> 172.21.15.150:0/597929234 conn(0x55ac32045a50 msgr2=0x55ac320477a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 shutdown_connections 2024-03-18T16:05:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 1 -- 172.21.15.150:0/597929234 wait complete. 2024-03-18T16:05:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.202+0000 7ff69d64c740 20 client.15864 trim_cache size 0 max 16384 2024-03-18T16:05:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 Processor -- start 2024-03-18T16:05:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 -- start start 2024-03-18T16:05:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32058300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac32058c30 0x55ac320475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941ada20 con 0x55ac3204b4f0 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941ab2f0 con 0x55ac3204abd0 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff69d64c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941a9180 con 0x55ac32058c30 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff698808640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32058300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff699009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac32058c30 0x55ac320475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49770/0 (socket says 172.21.15.150:49770) 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 -- 172.21.15.150:0/2089002264 learned_addr learned my addr 172.21.15.150:0/2089002264 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff698808640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32058300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:59054/0 (socket says 172.21.15.150:59054) 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 -- 172.21.15.150:0/2089002264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac32058c30 msgr2=0x55ac320475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac32058c30 0x55ac320475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 -- 172.21.15.150:0/2089002264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 msgr2=0x55ac32058300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32058300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 -- 172.21.15.150:0/2089002264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac3204b320 con 0x55ac3204b4f0 2024-03-18T16:05:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff693fff640 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff68002c820 tx=0x7ff68002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33f405a6e3a52d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.219+0000 7ff67e7fc640 1 -- 172.21.15.150:0/2089002264 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680054020 con 0x55ac3204b4f0 2024-03-18T16:05:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff67e7fc640 1 -- 172.21.15.150:0/2089002264 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff680005510 con 0x55ac3204b4f0 2024-03-18T16:05:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff67e7fc640 1 -- 172.21.15.150:0/2089002264 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800046c0 con 0x55ac3204b4f0 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 -- 172.21.15.150:0/2089002264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 msgr2=0x55ac32064580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff68002c820 tx=0x7ff68002bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 -- 172.21.15.150:0/2089002264 shutdown_connections 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac32058c30 0x55ac320475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac3204b4f0 0x55ac32064580 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 --2- 172.21.15.150:0/2089002264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32058300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 -- 172.21.15.150:0/2089002264 >> 172.21.15.150:0/2089002264 conn(0x55ac32045a50 msgr2=0x55ac32046ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 -- 172.21.15.150:0/2089002264 shutdown_connections 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.220+0000 7ff69d64c740 1 -- 172.21.15.150:0/2089002264 wait complete. 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 -- start start 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 Processor -- start 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32052fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204b4f0 0x55ac32053520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941a97d0 con 0x55ac32058c30 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941aac20 con 0x55ac3204abd0 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff6941a9e80 con 0x55ac3204b4f0 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff69d64c740 10 client.0 _ll_register_callbacks cb 0x7ff680009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff699009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff698808640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32052fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff699009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49782/0 (socket says 172.21.15.150:49782) 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff693fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204b4f0 0x55ac32053520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.221+0000 7ff699009640 1 -- 172.21.15.150:0/1475285615 learned_addr learned my addr 172.21.15.150:0/1475285615 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 -- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204b4f0 msgr2=0x55ac32053520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204b4f0 0x55ac32053520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 -- 172.21.15.150:0/1475285615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 msgr2=0x55ac320ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 -- 172.21.15.150:0/1475285615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac320ce780 con 0x55ac3204abd0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff699009640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff698808640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32052fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6880cc910 tx=0x7ff68800a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=717b5ab76d15dc31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688084070 con 0x55ac3204abd0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7ff6880979c0 con 0x55ac3204abd0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff680005510 con 0x55ac3204abd0 2024-03-18T16:05:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff68807b070 con 0x55ac3204abd0 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 10 client.25637 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac320cfa70 con 0x55ac3204abd0 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff680005510 con 0x55ac3204abd0 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ff68800b040 con 0x55ac3204abd0 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 10 client.25637 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 10 client.25637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 20 client.25637 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac320cfe00 0x55ac320d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 4 client.25637 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.222+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac31fb72d0 con 0x55ac320cfe00 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.223+0000 7ff693fff640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac320cfe00 0x55ac320d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:35.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.223+0000 7ff673fff640 10 client.25637 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:35.230 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.223+0000 7ff693fff640 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac320cfe00 0x55ac320d21f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b9c3b594dacac1 server_cookie=172fde7bb2b01d48 in_seq=0 out_seq=0 2024-03-18T16:05:35.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.231+0000 7ff673fff640 1 -- 172.21.15.150:0/1475285615 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac31fb72d0 con 0x55ac320cfe00 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.231+0000 7ff673fff640 10 client.25637 handle_command_reply: tid=0 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 client.25637 shutdown 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 msgr2=0x55ac32052fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32052fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6880cc910 tx=0x7ff68800a830 comp rx=0 tx=0).stop 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 shutdown_connections 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55ac320cfe00 0x55ac320d21f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ac32058c30 0x55ac320ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ac3204b4f0 0x55ac32053520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 --2- 172.21.15.150:0/1475285615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac3204abd0 0x55ac32052fe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff673fff640 0 client.25637 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 >> 172.21.15.150:0/1475285615 conn(0x55ac32045a50 msgr2=0x55ac32044bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 shutdown_connections 2024-03-18T16:05:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 1 -- 172.21.15.150:0/1475285615 wait complete. 2024-03-18T16:05:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69d64c740 20 client.25637 trim_cache size 0 max 16384 2024-03-18T16:05:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:05:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:05:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:05:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.232+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 msgr2=0x7ff6840a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 0x7ff6840a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff68c013c70 tx=0x7ff68c009e90 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 msgr2=0x7ff6941980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff68800ea20 tx=0x7ff68800eee0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 shutdown_connections 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6840a2a90 0x7ff6840a4f50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff69419c970 0x7ff69419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff694105130 0x7ff6941985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 --2- 172.21.15.150:0/533546229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100060 0x7ff6941980b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 >> 172.21.15.150:0/533546229 conn(0x7ff694076c70 msgr2=0x7ff6941038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 shutdown_connections 2024-03-18T16:05:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:35.233+0000 7ff69aa6c640 1 -- 172.21.15.150:0/533546229 wait complete. 2024-03-18T16:05:35.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 541805 11.76 MB/sec execute 140 sec latency 29.758 ms 2024-03-18T16:05:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:35 smithi191 ceph-mon[36784]: pgmap v169: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:35 smithi156 ceph-mon[35919]: pgmap v169: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:35 smithi150 ceph-mon[31946]: pgmap v169: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 213 op/s 2024-03-18T16:05:36.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 543963 11.78 MB/sec execute 141 sec latency 15.332 ms 2024-03-18T16:05:37.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 545889 11.74 MB/sec execute 142 sec latency 31.483 ms 2024-03-18T16:05:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:37 smithi191 ceph-mon[36784]: pgmap v170: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 315 op/s 2024-03-18T16:05:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:37 smithi156 ceph-mon[35919]: pgmap v170: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 315 op/s 2024-03-18T16:05:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:37 smithi150 ceph-mon[31946]: pgmap v170: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 315 op/s 2024-03-18T16:05:38.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 547489 11.75 MB/sec execute 143 sec latency 65.783 ms 2024-03-18T16:05:39.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 549408 11.71 MB/sec execute 144 sec latency 17.909 ms 2024-03-18T16:05:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:39 smithi191 ceph-mon[36784]: pgmap v171: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 207 op/s 2024-03-18T16:05:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:39 smithi156 ceph-mon[35919]: pgmap v171: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 207 op/s 2024-03-18T16:05:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:39 smithi150 ceph-mon[31946]: pgmap v171: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.6 MiB/s wr, 207 op/s 2024-03-18T16:05:40.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 551233 11.72 MB/sec execute 145 sec latency 29.009 ms 2024-03-18T16:05:41.325 INFO:tasks.workunit.client.0.smithi150.stdout: 1 553196 11.69 MB/sec execute 146 sec latency 21.608 ms 2024-03-18T16:05:41.966 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:41 smithi191 ceph-mon[36784]: pgmap v172: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.1 MiB/s wr, 310 op/s 2024-03-18T16:05:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:41 smithi156 ceph-mon[35919]: pgmap v172: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.1 MiB/s wr, 310 op/s 2024-03-18T16:05:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:41 smithi150 ceph-mon[31946]: pgmap v172: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.1 MiB/s wr, 310 op/s 2024-03-18T16:05:42.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 554987 11.69 MB/sec execute 147 sec latency 22.531 ms 2024-03-18T16:05:43.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 556718 11.66 MB/sec execute 148 sec latency 60.544 ms 2024-03-18T16:05:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:43 smithi191 ceph-mon[36784]: pgmap v173: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:43 smithi156 ceph-mon[35919]: pgmap v173: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:43 smithi150 ceph-mon[31946]: pgmap v173: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:44.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 558573 11.66 MB/sec execute 149 sec latency 17.806 ms 2024-03-18T16:05:45.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 560694 11.66 MB/sec execute 150 sec latency 17.218 ms 2024-03-18T16:05:45.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:45 smithi191 ceph-mon[36784]: pgmap v174: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:45 smithi156 ceph-mon[35919]: pgmap v174: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:45 smithi150 ceph-mon[31946]: pgmap v174: 97 pgs: 97 active+clean; 338 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-18T16:05:46.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 562437 11.64 MB/sec execute 151 sec latency 18.374 ms 2024-03-18T16:05:47.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 564779 11.67 MB/sec execute 152 sec latency 19.542 ms 2024-03-18T16:05:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:47 smithi191 ceph-mon[36784]: pgmap v175: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-18T16:05:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:47 smithi156 ceph-mon[35919]: pgmap v175: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-18T16:05:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:47 smithi150 ceph-mon[31946]: pgmap v175: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 300 op/s 2024-03-18T16:05:48.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 566032 11.62 MB/sec execute 153 sec latency 110.196 ms 2024-03-18T16:05:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 -- 172.21.15.150:0/2084519267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e04105680 msgr2=0x7f7e04105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 --2- 172.21.15.150:0/2084519267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e04105680 0x7f7e04105a60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009900 tx=0x7f7e0002bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 -- 172.21.15.150:0/2084519267 shutdown_connections 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 --2- 172.21.15.150:0/2084519267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e041012f0 0x7f7e0410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 --2- 172.21.15.150:0/2084519267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04100930 0x7f7e04100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 --2- 172.21.15.150:0/2084519267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e04105680 0x7f7e04105a60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 -- 172.21.15.150:0/2084519267 >> 172.21.15.150:0/2084519267 conn(0x7f7e04077530 msgr2=0x7f7e04077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 -- 172.21.15.150:0/2084519267 shutdown_connections 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.324+0000 7f7e0c377640 1 -- 172.21.15.150:0/2084519267 wait complete. 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 Processor -- start 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 -- start start 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 0x7f7e04198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04105680 0x7f7e041a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e04110ba0 con 0x7f7e041012f0 2024-03-18T16:05:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e04110a20 con 0x7f7e04105680 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0c377640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7e04110d20 con 0x7f7e04100930 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e09912640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 0x7f7e04198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e09912640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38782/0 (socket says 172.21.15.150:38782) 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e09912640 1 -- 172.21.15.150:0/3183986604 learned_addr learned my addr 172.21.15.150:0/3183986604 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.325+0000 7f7e0a914640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04105680 0x7f7e041a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 -- 172.21.15.150:0/3183986604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04105680 msgr2=0x7f7e041a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04105680 0x7f7e041a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 -- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 msgr2=0x7f7e04199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 -- 172.21.15.150:0/3183986604 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e041a44c0 con 0x7f7e04100930 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e09912640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 0x7f7e04198f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e00000c00 tx=0x7f7e000042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7dfb7fe640 1 -- 172.21.15.150:0/3183986604 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00041070 con 0x7f7e04100930 2024-03-18T16:05:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e041a4690 con 0x7f7e04100930 2024-03-18T16:05:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7dfb7fe640 1 -- 172.21.15.150:0/3183986604 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e0002ccc0 con 0x7f7e04100930 2024-03-18T16:05:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7dfb7fe640 1 -- 172.21.15.150:0/3183986604 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00003db0 con 0x7f7e04100930 2024-03-18T16:05:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.326+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e041a4af0 con 0x7f7e04100930 2024-03-18T16:05:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.328+0000 7f7dfb7fe640 1 -- 172.21.15.150:0/3183986604 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f7e00044070 con 0x7f7e04100930 2024-03-18T16:05:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.329+0000 7f7dfb7fe640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 0x7f7df00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.329+0000 7f7e0ef63740 1 Processor -- start 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.329+0000 7f7dfb7fe640 1 -- 172.21.15.150:0/3183986604 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f7e000f06d0 con 0x7f7e04100930 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.329+0000 7f7e09912640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 0x7f7df00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 -- start start 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20d7dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563c20cd6a40 con 0x563c20d72130 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c20c7f920 con 0x563c20d7b820 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c20cd2ae0 con 0x563c20d72a50 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e09912640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 0x7f7df00a4ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e04069960 tx=0x7f7df4005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38794/0 (socket says 172.21.15.150:38794) 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 -- 172.21.15.150:0/4100186362 learned_addr learned my addr 172.21.15.150:0/4100186362 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a914640 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20d7dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e09912640 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 -- 172.21.15.150:0/4100186362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 msgr2=0x563c20d7b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 -- 172.21.15.150:0/4100186362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 msgr2=0x563c20d7dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20d7dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 -- 172.21.15.150:0/4100186362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c20c97460 con 0x563c20d72130 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7e0a113640 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b9370 tx=0x7f7e000038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde3e25880b5870d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.330+0000 7f7df8bfe640 1 -- 172.21.15.150:0/4100186362 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0004b040 con 0x563c20d72130 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7df8bfe640 1 -- 172.21.15.150:0/4100186362 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e00005670 con 0x563c20d72130 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7df8bfe640 1 -- 172.21.15.150:0/4100186362 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e000b9580 con 0x563c20d72130 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 -- 172.21.15.150:0/4100186362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 msgr2=0x563c20d72510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b9370 tx=0x7f7e000038b0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 -- 172.21.15.150:0/4100186362 shutdown_connections 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20d7dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/4100186362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d72510 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 -- 172.21.15.150:0/4100186362 >> 172.21.15.150:0/4100186362 conn(0x563c20d6ca50 msgr2=0x563c20d6ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 -- 172.21.15.150:0/4100186362 shutdown_connections 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.331+0000 7f7e0ef63740 1 -- 172.21.15.150:0/4100186362 wait complete. 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 -- start start 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 Processor -- start 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d78e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x563c20d7e210 con 0x563c20d7b820 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563c20c92d70 con 0x563c20d72a50 2024-03-18T16:05:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x563c20d7e390 con 0x563c20d72130 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0ef63740 10 client.0 _ll_register_callbacks cb 0x563c209eaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38800/0 (socket says 172.21.15.150:38800) 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 -- 172.21.15.150:0/2257922891 learned_addr learned my addr 172.21.15.150:0/2257922891 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a113640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e09912640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d78e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 -- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 msgr2=0x563c20d788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d788e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 -- 172.21.15.150:0/2257922891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 msgr2=0x563c20d78e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d78e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.332+0000 7f7e0a914640 1 -- 172.21.15.150:0/2257922891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c209eb210 con 0x563c20d7b820 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0a914640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc011cf0 tx=0x7f7dfc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d24eb8ef168328 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00fe60 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f7dfc003970 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e00005670 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7dfc003b70 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 10 client.15882 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c20ab7620 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e00005670 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f7dfc022070 con 0x563c20d7b820 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 10 client.15882 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 20 client.15882 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20d8b840 0x563c20d8dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 4 client.15882 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:05:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.333+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c20ce3f80 con 0x563c20d8b840 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.334+0000 7f7e0a113640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20d8b840 0x563c20d8dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.334+0000 7f7e0a113640 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20d8b840 0x563c20d8dc30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6994b5679e2c60 server_cookie=b288ea772b916fb8 in_seq=0 out_seq=0 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.334+0000 7f7dedffb640 10 client.15882 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7dedffb640 1 -- 172.21.15.150:0/2257922891 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c20ce3f80 con 0x563c20d8b840 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7dedffb640 10 client.15882 handle_command_reply: tid=0 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 client.15882 shutdown 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 msgr2=0x563c20d79360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc011cf0 tx=0x7f7dfc00f790 comp rx=0 tx=0).stop 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 shutdown_connections 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20d8b840 0x563c20d8dc30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d79360 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d78e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2257922891 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7dedffb640 0 client.15882 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 >> 172.21.15.150:0/2257922891 conn(0x563c20d6ca50 msgr2=0x563c20d6e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 shutdown_connections 2024-03-18T16:05:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2257922891 wait complete. 2024-03-18T16:05:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.335+0000 7f7e0ef63740 20 client.15882 trim_cache size 0 max 16384 2024-03-18T16:05:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 Processor -- start 2024-03-18T16:05:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 -- start start 2024-03-18T16:05:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d7b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041afc50 con 0x563c20d7b820 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041ad520 con 0x563c20d72a50 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041ab370 con 0x563c20d72130 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0a914640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d7b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33364/0 (socket says 172.21.15.150:33364) 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0a914640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38810/0 (socket says 172.21.15.150:38810) 2024-03-18T16:05:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 -- 172.21.15.150:0/2125026712 learned_addr learned my addr 172.21.15.150:0/2125026712 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 -- 172.21.15.150:0/2125026712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 msgr2=0x563c20d7b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d7b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 -- 172.21.15.150:0/2125026712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 msgr2=0x563c20d6bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 -- 172.21.15.150:0/2125026712 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c20d77890 con 0x563c20d72a50 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e0a914640 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.350+0000 7f7e09912640 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009fd0 tx=0x7f7df4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1559bfa7190432d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7deffff640 1 -- 172.21.15.150:0/2125026712 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df402e070 con 0x563c20d72a50 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7deffff640 1 -- 172.21.15.150:0/2125026712 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df4029b20 con 0x563c20d72a50 2024-03-18T16:05:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7deffff640 1 -- 172.21.15.150:0/2125026712 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df401cc00 con 0x563c20d72a50 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2125026712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 msgr2=0x563c20d86ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009fd0 tx=0x7f7df4018000 comp rx=0 tx=0).stop 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2125026712 shutdown_connections 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d7b820 0x563c20d6bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d72a50 0x563c20d86ee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/2125026712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72130 0x563c20d7b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2125026712 >> 172.21.15.150:0/2125026712 conn(0x563c20d6ca50 msgr2=0x563c20d6ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2125026712 shutdown_connections 2024-03-18T16:05:48.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.351+0000 7f7e0ef63740 1 -- 172.21.15.150:0/2125026712 wait complete. 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 -- start start 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 Processor -- start 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20df0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041ab9e0 con 0x563c20d72130 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041ace50 con 0x563c20d7b820 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7e041ac0b0 con 0x563c20d72a50 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0ef63740 10 client.0 _ll_register_callbacks cb 0x7f7dfc009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38818/0 (socket says 172.21.15.150:38818) 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 -- 172.21.15.150:0/3898407759 learned_addr learned my addr 172.21.15.150:0/3898407759 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a914640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20df0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e09912640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 -- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 msgr2=0x563c20d7fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 -- 172.21.15.150:0/3898407759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 msgr2=0x563c20df0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20df0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 -- 172.21.15.150:0/3898407759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c20d72880 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7e0a113640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b98e0 tx=0x7f7e00009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c5251c34fc66f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.352+0000 7f7dedffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0004b040 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f7e000ee030 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df4029b20 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7e000ee230 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 10 client.15888 fetch_fsmap learned FSMap version 29 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c20df22a0 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df4029b20 con 0x563c20d72130 2024-03-18T16:05:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f7e000c5170 con 0x563c20d72130 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 10 client.15888 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 20 client.15888 populate_metadata read hostname 'smithi150' 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20df2810 0x563c20df4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 4 client.15888 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c20cde2d0 con 0x563c20df2810 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.353+0000 7f7e09912640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20df2810 0x563c20df4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.354+0000 7f7dedffb640 10 client.15888 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.354+0000 7f7e09912640 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20df2810 0x563c20df4c30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac00f41e354825a server_cookie=986d46f17ae67e34 in_seq=0 out_seq=0 2024-03-18T16:05:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7dedffb640 1 -- 172.21.15.150:0/3898407759 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563c20cde2d0 con 0x563c20df2810 2024-03-18T16:05:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7dedffb640 10 client.15888 handle_command_reply: tid=0 2024-03-18T16:05:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 client.15888 shutdown 2024-03-18T16:05:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 msgr2=0x563c20d7f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7e000b98e0 tx=0x7f7e00009680 comp rx=0 tx=0).stop 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 shutdown_connections 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x563c20df2810 0x563c20df4c30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563c20d7b820 0x563c20df0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x563c20d72a50 0x563c20d7fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 --2- 172.21.15.150:0/3898407759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563c20d72130 0x563c20d7f5a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 >> 172.21.15.150:0/3898407759 conn(0x563c20d6ca50 msgr2=0x563c20d79110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7dedffb640 0 client.15888 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 shutdown_connections 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.363+0000 7f7e0ef63740 1 -- 172.21.15.150:0/3898407759 wait complete. 2024-03-18T16:05:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0ef63740 20 client.15888 trim_cache size 0 max 16384 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 msgr2=0x7f7df00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 0x7f7df00a4ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e04069960 tx=0x7f7df4005f90 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 msgr2=0x7f7e04198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 0x7f7e04198f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e00000c00 tx=0x7f7e000042e0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 shutdown_connections 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7df00a2a10 0x7f7df00a4ed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e04105680 0x7f7e041a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e041012f0 0x7f7e04199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 --2- 172.21.15.150:0/3183986604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e04100930 0x7f7e04198f00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 >> 172.21.15.150:0/3183986604 conn(0x7f7e04077530 msgr2=0x7f7e040fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 shutdown_connections 2024-03-18T16:05:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:05:48.364+0000 7f7e0c377640 1 -- 172.21.15.150:0/3183986604 wait complete. 2024-03-18T16:05:49.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 568339 11.64 MB/sec execute 154 sec latency 28.902 ms 2024-03-18T16:05:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:49 smithi191 ceph-mon[36784]: pgmap v176: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 198 op/s 2024-03-18T16:05:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:49 smithi156 ceph-mon[35919]: pgmap v176: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 198 op/s 2024-03-18T16:05:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:49 smithi150 ceph-mon[31946]: pgmap v176: 97 pgs: 97 active+clean; 344 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 198 op/s 2024-03-18T16:05:50.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 569863 11.61 MB/sec execute 155 sec latency 22.551 ms 2024-03-18T16:05:51.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 572125 11.62 MB/sec execute 156 sec latency 16.484 ms 2024-03-18T16:05:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:51 smithi191 ceph-mon[36784]: pgmap v177: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 297 op/s 2024-03-18T16:05:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:51 smithi156 ceph-mon[35919]: pgmap v177: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 297 op/s 2024-03-18T16:05:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:51 smithi150 ceph-mon[31946]: pgmap v177: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 297 op/s 2024-03-18T16:05:52.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 573546 11.58 MB/sec execute 157 sec latency 29.381 ms 2024-03-18T16:05:53.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 575660 11.60 MB/sec execute 158 sec latency 91.294 ms 2024-03-18T16:05:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:53 smithi191 ceph-mon[36784]: pgmap v178: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:53 smithi156 ceph-mon[35919]: pgmap v178: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:53 smithi150 ceph-mon[31946]: pgmap v178: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:54.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 577210 11.56 MB/sec execute 159 sec latency 20.139 ms 2024-03-18T16:05:55.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 579412 11.57 MB/sec execute 160 sec latency 17.268 ms 2024-03-18T16:05:55.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:55 smithi191 ceph-mon[36784]: pgmap v179: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:55 smithi156 ceph-mon[35919]: pgmap v179: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:55 smithi150 ceph-mon[31946]: pgmap v179: 97 pgs: 97 active+clean; 350 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-03-18T16:05:56.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 581323 11.54 MB/sec execute 161 sec latency 35.593 ms 2024-03-18T16:05:57.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 583119 11.55 MB/sec execute 162 sec latency 16.871 ms 2024-03-18T16:05:57.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:57 smithi191 ceph-mon[36784]: pgmap v180: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 289 op/s 2024-03-18T16:05:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:57 smithi156 ceph-mon[35919]: pgmap v180: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 289 op/s 2024-03-18T16:05:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:57 smithi150 ceph-mon[31946]: pgmap v180: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 289 op/s 2024-03-18T16:05:58.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 584373 11.51 MB/sec execute 163 sec latency 145.148 ms 2024-03-18T16:05:59.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 586498 11.52 MB/sec execute 164 sec latency 17.595 ms 2024-03-18T16:05:59.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:05:59 smithi191 ceph-mon[36784]: pgmap v181: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-18T16:06:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:05:59 smithi156 ceph-mon[35919]: pgmap v181: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-18T16:06:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:05:59 smithi150 ceph-mon[31946]: pgmap v181: 97 pgs: 97 active+clean; 356 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 193 op/s 2024-03-18T16:06:00.326 INFO:tasks.workunit.client.0.smithi150.stdout: 1 588461 11.50 MB/sec execute 165 sec latency 19.335 ms 2024-03-18T16:06:01.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 589946 11.50 MB/sec execute 166 sec latency 22.462 ms 2024-03-18T16:06:01.456 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 -- 172.21.15.150:0/1144524535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c1078b0 msgr2=0x7fd44c107c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 --2- 172.21.15.150:0/1144524535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c1078b0 0x7fd44c107c90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd440009930 tx=0x7fd44002bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 -- 172.21.15.150:0/1144524535 shutdown_connections 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 --2- 172.21.15.150:0/1144524535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c069800 0x7fd44c10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 --2- 172.21.15.150:0/1144524535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c068e60 0x7fd44c0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 --2- 172.21.15.150:0/1144524535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c1078b0 0x7fd44c107c90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 -- 172.21.15.150:0/1144524535 >> 172.21.15.150:0/1144524535 conn(0x7fd44c077530 msgr2=0x7fd44c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 -- 172.21.15.150:0/1144524535 shutdown_connections 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.452+0000 7fd452e34640 1 -- 172.21.15.150:0/1144524535 wait complete. 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 Processor -- start 2024-03-18T16:06:01.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 -- start start 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c068e60 0x7fd44c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c1078b0 0x7fd44c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111ba0 con 0x7fd44c069800 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111a20 con 0x7fd44c068e60 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd452e34640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111d20 con 0x7fd44c1078b0 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54650/0 (socket says 172.21.15.150:54650) 2024-03-18T16:06:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 -- 172.21.15.150:0/4291877914 learned_addr learned my addr 172.21.15.150:0/4291877914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd450bd0640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c068e60 0x7fd44c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd4513d1640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c1078b0 0x7fd44c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 -- 172.21.15.150:0/4291877914 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c1078b0 msgr2=0x7fd44c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c1078b0 0x7fd44c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 -- 172.21.15.150:0/4291877914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c068e60 msgr2=0x7fd44c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c068e60 0x7fd44c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.453+0000 7fd44bfff640 1 -- 172.21.15.150:0/4291877914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c19cfd0 con 0x7fd44c069800 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd43800e9f0 tx=0x7fd43800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd449ffb640 1 -- 172.21.15.150:0/4291877914 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43800cd80 con 0x7fd44c069800 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd449ffb640 1 -- 172.21.15.150:0/4291877914 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd438005500 con 0x7fd44c069800 2024-03-18T16:06:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c19d2c0 con 0x7fd44c069800 2024-03-18T16:06:01.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd449ffb640 1 -- 172.21.15.150:0/4291877914 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438010690 con 0x7fd44c069800 2024-03-18T16:06:01.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.454+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c1a4b50 con 0x7fd44c069800 2024-03-18T16:06:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.456+0000 7fd449ffb640 1 -- 172.21.15.150:0/4291877914 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fd4380046c0 con 0x7fd44c069800 2024-03-18T16:06:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.456+0000 7fd455a19740 1 Processor -- start 2024-03-18T16:06:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.456+0000 7fd449ffb640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 0x7fd43c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.456+0000 7fd455a19740 1 -- start start 2024-03-18T16:06:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd4513d1640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 0x7fd43c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd449ffb640 1 -- 172.21.15.150:0/4291877914 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fd4380cf230 con 0x7fd44c069800 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823130 0x55d9f5823510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f582c820 0x55d9f582ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d9f5787a40 con 0x55d9f582c820 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d9f5730920 con 0x55d9f5823a50 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd455a19740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d9f5783ae0 con 0x55d9f5823130 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823130 0x55d9f5823510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:37984/0 (socket says 172.21.15.150:37984) 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 -- 172.21.15.150:0/4219629874 learned_addr learned my addr 172.21.15.150:0/4219629874 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd4513d1640 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f582c820 0x55d9f582ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 -- 172.21.15.150:0/4219629874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823130 msgr2=0x55d9f5823510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823130 0x55d9f5823510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd4513d1640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 0x7fd43c0a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd444011810 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 -- 172.21.15.150:0/4219629874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f582c820 msgr2=0x55d9f582ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f582c820 0x55d9f582ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 -- 172.21.15.150:0/4219629874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f5748460 con 0x55d9f5823a50 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd44bfff640 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd438009380 tx=0x7fd43809b210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91919515136137ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd436ffd640 1 -- 172.21.15.150:0/4219629874 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43809fa00 con 0x55d9f5823a50 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd436ffd640 1 -- 172.21.15.150:0/4219629874 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4380a4050 con 0x55d9f5823a50 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.457+0000 7fd436ffd640 1 -- 172.21.15.150:0/4219629874 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43809ce40 con 0x55d9f5823a50 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 -- 172.21.15.150:0/4219629874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 msgr2=0x55d9f582c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd438009380 tx=0x7fd43809b210 comp rx=0 tx=0).stop 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 -- 172.21.15.150:0/4219629874 shutdown_connections 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f582c820 0x55d9f582ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f582c2e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 --2- 172.21.15.150:0/4219629874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823130 0x55d9f5823510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 -- 172.21.15.150:0/4219629874 >> 172.21.15.150:0/4219629874 conn(0x55d9f581da50 msgr2=0x55d9f581de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 -- 172.21.15.150:0/4219629874 shutdown_connections 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.458+0000 7fd455a19740 1 -- 172.21.15.150:0/4219629874 wait complete. 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 -- start start 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 Processor -- start 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823a50 0x55d9f5829df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 0x55d9f582a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d9f582f450 con 0x55d9f5823130 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d9f5743d70 con 0x55d9f582c820 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d9f582f5d0 con 0x55d9f5823a50 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd455a19740 10 client.0 _ll_register_callbacks cb 0x55d9f549ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54674/0 (socket says 172.21.15.150:54674) 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 -- 172.21.15.150:0/1030003703 learned_addr learned my addr 172.21.15.150:0/1030003703 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd4513d1640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 0x55d9f582a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd44bfff640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823a50 0x55d9f5829df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.472 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 -- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823a50 msgr2=0x55d9f5829df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823a50 0x55d9f5829df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 -- 172.21.15.150:0/1030003703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 msgr2=0x55d9f582a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 0x55d9f582a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd450bd0640 1 -- 172.21.15.150:0/1030003703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f549c210 con 0x55d9f5823130 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.459+0000 7fd4513d1640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 0x55d9f582a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44002c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cc846dc8e3d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44002ccc0 con 0x55d9f5823130 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fd4400046c0 con 0x55d9f5823130 2024-03-18T16:06:01.473 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4380a4050 con 0x55d9f5823130 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd4400048c0 con 0x55d9f5823130 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 10 client.15900 fetch_fsmap learned FSMap version 29 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9f5568620 con 0x55d9f5823130 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4380a4050 con 0x55d9f5823130 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd44004e070 con 0x55d9f5823130 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 10 client.15900 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 20 client.15900 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f583ca20 0x55d9f583ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 4 client.15900 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.460+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9f5794f80 con 0x55d9f583ca20 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.461+0000 7fd44bfff640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f583ca20 0x55d9f583ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.461+0000 7fd44bfff640 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f583ca20 0x55d9f583ee30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f25fc6031222010 server_cookie=7f516d75788e8f4e in_seq=0 out_seq=0 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.461+0000 7fd42bfff640 10 client.15900 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd42bfff640 1 -- 172.21.15.150:0/1030003703 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9f5794f80 con 0x55d9f583ca20 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd42bfff640 10 client.15900 handle_command_reply: tid=0 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 client.15900 shutdown 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 msgr2=0x55d9f58298b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44002c800 comp rx=0 tx=0).stop 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 shutdown_connections 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f583ca20 0x55d9f583ee30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f582c820 0x55d9f582a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd42bfff640 0 client.15900 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f5823a50 0x55d9f5829df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 --2- 172.21.15.150:0/1030003703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58298b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 >> 172.21.15.150:0/1030003703 conn(0x55d9f581da50 msgr2=0x55d9f581f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 shutdown_connections 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 1 -- 172.21.15.150:0/1030003703 wait complete. 2024-03-18T16:06:01.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.462+0000 7fd455a19740 20 client.15900 trim_cache size 0 max 16384 2024-03-18T16:06:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 Processor -- start 2024-03-18T16:06:01.484 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 -- start start 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f5844740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f581eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1afc50 con 0x55d9f5823130 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ad520 con 0x55d9f5823a50 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd455a19740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ab370 con 0x55d9f582c820 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54680/0 (socket says 172.21.15.150:54680) 2024-03-18T16:06:01.485 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 -- 172.21.15.150:0/1193429189 learned_addr learned my addr 172.21.15.150:0/1193429189 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd4513d1640 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f581eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd44bfff640 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f5844740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 -- 172.21.15.150:0/1193429189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 msgr2=0x55d9f581eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f581eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 -- 172.21.15.150:0/1193429189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 msgr2=0x55d9f5844740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f5844740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.477+0000 7fd450bd0640 1 -- 172.21.15.150:0/1193429189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f5817fc0 con 0x55d9f5823130 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd450bd0640 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd440005aa0 tx=0x7fd440004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f276690fae0117d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd4367fc640 1 -- 172.21.15.150:0/1193429189 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440054070 con 0x55d9f5823130 2024-03-18T16:06:01.486 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd4367fc640 1 -- 172.21.15.150:0/1193429189 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd440009520 con 0x55d9f5823130 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd4367fc640 1 -- 172.21.15.150:0/1193429189 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440004f10 con 0x55d9f5823130 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 -- 172.21.15.150:0/1193429189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 msgr2=0x55d9f58381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd440005aa0 tx=0x7fd440004500 comp rx=0 tx=0).stop 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 -- 172.21.15.150:0/1193429189 shutdown_connections 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f581eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823a50 0x55d9f5844740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 --2- 172.21.15.150:0/1193429189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823130 0x55d9f58381f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 -- 172.21.15.150:0/1193429189 >> 172.21.15.150:0/1193429189 conn(0x55d9f581da50 msgr2=0x55d9f5818a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 -- 172.21.15.150:0/1193429189 shutdown_connections 2024-03-18T16:06:01.487 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.478+0000 7fd455a19740 1 -- 172.21.15.150:0/1193429189 wait complete. 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 -- start start 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 Processor -- start 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 0x55d9f5826580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f5827000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ab9e0 con 0x55d9f5823a50 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ace50 con 0x55d9f5823130 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c1ac0b0 con 0x55d9f582c820 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd455a19740 10 client.0 _ll_register_callbacks cb 0x7fd440009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54682/0 (socket says 172.21.15.150:54682) 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd4513d1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f5827000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 -- 172.21.15.150:0/676992044 learned_addr learned my addr 172.21.15.150:0/676992044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd450bd0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 0x55d9f5826580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 -- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 msgr2=0x55d9f5827000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f5827000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.479+0000 7fd44bfff640 1 -- 172.21.15.150:0/676992044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 msgr2=0x55d9f5826580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd44bfff640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 0x55d9f5826580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd44bfff640 1 -- 172.21.15.150:0/676992044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f58a1f20 con 0x55d9f5823a50 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd450bd0640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 0x55d9f5826580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:01.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd44bfff640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd4380ccfa0 tx=0x7fd43809ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c299a69ce1eedc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4380a4050 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7fd438097f20 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd440009520 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd4380ccc20 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 10 client.15906 fetch_fsmap learned FSMap version 29 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9f58a31f0 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd440009520 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fd4380a6060 con 0x55d9f5823a50 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 10 client.15906 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 10 client.15906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:01.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 20 client.15906 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f58a34f0 0x55d9f58a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 4 client.15906 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.480+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9f578f2d0 con 0x55d9f58a34f0 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.481+0000 7fd450bd0640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f58a34f0 0x55d9f58a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.481+0000 7fd42bfff640 10 client.15906 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:01.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:01.481+0000 7fd450bd0640 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f58a34f0 0x55d9f58a58e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d90693391957c04 server_cookie=828e2f25ff3c6472 in_seq=0 out_seq=0 2024-03-18T16:06:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:01 smithi191 ceph-mon[36784]: pgmap v182: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-18T16:06:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:01 smithi156 ceph-mon[35919]: pgmap v182: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-18T16:06:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:01 smithi150 ceph-mon[31946]: pgmap v182: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.166+0000 7fd42bfff640 1 -- 172.21.15.150:0/676992044 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9f578f2d0 con 0x55d9f58a34f0 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.166+0000 7fd42bfff640 10 client.15906 handle_command_reply: tid=0 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.166+0000 7fd455a19740 1 client.15906 shutdown 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.166+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 msgr2=0x55d9f5826ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.166+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd4380ccfa0 tx=0x7fd43809ba30 comp rx=0 tx=0).stop 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 shutdown_connections 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9f58a34f0 0x55d9f58a58e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d9f582c820 0x55d9f5827000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d9f5823a50 0x55d9f5826ac0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd42bfff640 0 client.15906 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 --2- 172.21.15.150:0/676992044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9f5823130 0x55d9f5826580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 >> 172.21.15.150:0/676992044 conn(0x55d9f581da50 msgr2=0x55d9f581c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 shutdown_connections 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 1 -- 172.21.15.150:0/676992044 wait complete. 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd455a19740 20 client.15906 trim_cache size 0 max 16384 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:06:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 msgr2=0x7fd43c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 0x7fd43c0a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd444011810 comp rx=0 tx=0).stop 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 msgr2=0x7fd44c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd43800e9f0 tx=0x7fd43800edf0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 shutdown_connections 2024-03-18T16:06:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd43c0a2a10 0x7fd43c0a4ed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd44c1078b0 0x7fd44c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd44c069800 0x7fd44c198520 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 --2- 172.21.15.150:0/4291877914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd44c068e60 0x7fd44c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 >> 172.21.15.150:0/4291877914 conn(0x7fd44c077530 msgr2=0x7fd44c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.167+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 shutdown_connections 2024-03-18T16:06:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:02.168+0000 7fd452e34640 1 -- 172.21.15.150:0/4291877914 wait complete. 2024-03-18T16:06:02.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 591399 11.46 MB/sec execute 167 sec latency 31.165 ms 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: *** Caught signal (Segmentation fault) ** 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: in thread 7fab74997700 thread_name:md_log_replay 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fab85d58d20] 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2: /usr/bin/ceph-mds(+0x548b3f) [0x56475df32b3f] 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x56475df362d3] 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 4: (MDLog::_replay_thread()+0x617) [0x56475df8fd37] 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 5: (MDLog::ReplayThread::entry()+0x11) [0x56475df99421] 2024-03-18T16:06:03.215 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fab85d4e1ca] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 7: clone() 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:02.883+0000 7fab74997700 -1 *** Caught signal (Segmentation fault) ** 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: in thread 7fab74997700 thread_name:md_log_replay 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fab85d58d20] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2: /usr/bin/ceph-mds(+0x548b3f) [0x56475df32b3f] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x56475df362d3] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 4: (MDLog::_replay_thread()+0x617) [0x56475df8fd37] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 5: (MDLog::ReplayThread::entry()+0x11) [0x56475df99421] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fab85d4e1ca] 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 7: clone() 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-18T16:06:03.216 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 0> 2024-03-18T16:06:02.883+0000 7fab74997700 -1 *** Caught signal (Segmentation fault) ** 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: in thread 7fab74997700 thread_name:md_log_replay 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fab85d58d20] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2: /usr/bin/ceph-mds(+0x548b3f) [0x56475df32b3f] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x56475df362d3] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 4: (MDLog::_replay_thread()+0x617) [0x56475df8fd37] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 5: (MDLog::ReplayThread::entry()+0x11) [0x56475df99421] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fab85d4e1ca] 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 7: clone() 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 0> 2024-03-18T16:06:02.883+0000 7fab74997700 -1 *** Caught signal (Segmentation fault) ** 2024-03-18T16:06:03.217 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: in thread 7fab74997700 thread_name:md_log_replay 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fab85d58d20] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2: /usr/bin/ceph-mds(+0x548b3f) [0x56475df32b3f] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x56475df362d3] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 4: (MDLog::_replay_thread()+0x617) [0x56475df8fd37] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 5: (MDLog::ReplayThread::entry()+0x11) [0x56475df99421] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fab85d4e1ca] 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 7: clone() 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[67308]: 2024-03-18T16:06:03.218 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:02 smithi191 podman[71903]: 2024-03-18 16:06:02.95900574 +0000 UTC m=+0.023828416 container died 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T16:06:03.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 593403 11.48 MB/sec execute 168 sec latency 111.821 ms 2024-03-18T16:06:03.935 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:03 smithi191 ceph-mon[36784]: pgmap v183: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:03 smithi156 ceph-mon[35919]: pgmap v183: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:03 smithi150 ceph-mon[31946]: pgmap v183: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:04.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 594755 11.44 MB/sec execute 169 sec latency 130.720 ms 2024-03-18T16:06:05.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 596825 11.45 MB/sec execute 170 sec latency 55.385 ms 2024-03-18T16:06:05.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:05 smithi191 ceph-mon[36784]: pgmap v184: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:05 smithi156 ceph-mon[35919]: pgmap v184: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:05.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:05 smithi150 ceph-mon[31946]: pgmap v184: 97 pgs: 97 active+clean; 363 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-18T16:06:06.119 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:05 smithi191 podman[71903]: 2024-03-18 16:06:05.813059254 +0000 UTC m=+2.877881901 container remove 908b684a7ed2245b8c3b5a2fbe0d1d4a39bada713315fe5493853e51d1a8a0d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:06:06.119 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:05 smithi191 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service: Main process exited, code=exited, status=139/n/a 2024-03-18T16:06:06.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 598121 11.42 MB/sec execute 171 sec latency 100.112 ms 2024-03-18T16:06:06.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:06 smithi191 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-18T16:06:06.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:06 smithi191 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service: Consumed 1.476s CPU time. 2024-03-18T16:06:07.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 600373 11.42 MB/sec execute 172 sec latency 21.349 ms 2024-03-18T16:06:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:07 smithi191 ceph-mon[36784]: pgmap v185: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 320 op/s 2024-03-18T16:06:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:07 smithi156 ceph-mon[35919]: pgmap v185: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 320 op/s 2024-03-18T16:06:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:07 smithi150 ceph-mon[31946]: pgmap v185: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 320 op/s 2024-03-18T16:06:08.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 601680 11.40 MB/sec execute 173 sec latency 127.624 ms 2024-03-18T16:06:09.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 603916 11.40 MB/sec execute 174 sec latency 17.533 ms 2024-03-18T16:06:09.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:09 smithi191 ceph-mon[36784]: pgmap v186: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 224 op/s 2024-03-18T16:06:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:09 smithi156 ceph-mon[35919]: pgmap v186: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 224 op/s 2024-03-18T16:06:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:09 smithi150 ceph-mon[31946]: pgmap v186: 97 pgs: 97 active+clean; 369 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 224 op/s 2024-03-18T16:06:10.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 605391 11.38 MB/sec execute 175 sec latency 24.912 ms 2024-03-18T16:06:11.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 607637 11.39 MB/sec execute 176 sec latency 17.609 ms 2024-03-18T16:06:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:11 smithi191 ceph-mon[36784]: pgmap v187: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 321 op/s 2024-03-18T16:06:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:11 smithi156 ceph-mon[35919]: pgmap v187: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 321 op/s 2024-03-18T16:06:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:11 smithi150 ceph-mon[31946]: pgmap v187: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 321 op/s 2024-03-18T16:06:12.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 609012 11.36 MB/sec execute 177 sec latency 31.767 ms 2024-03-18T16:06:13.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 611143 11.37 MB/sec execute 178 sec latency 41.322 ms 2024-03-18T16:06:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:13 smithi191 ceph-mon[36784]: pgmap v188: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:13 smithi156 ceph-mon[35919]: pgmap v188: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:13 smithi150 ceph-mon[31946]: pgmap v188: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:14.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 612453 11.33 MB/sec execute 179 sec latency 87.511 ms 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 -- 172.21.15.150:0/1108395575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c105680 msgr2=0x7f179c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 --2- 172.21.15.150:0/1108395575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c105680 0x7f179c105a60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1798009900 tx=0x7f179802bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 -- 172.21.15.150:0/1108395575 shutdown_connections 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 --2- 172.21.15.150:0/1108395575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c1012f0 0x7f179c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 --2- 172.21.15.150:0/1108395575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c100930 0x7f179c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 --2- 172.21.15.150:0/1108395575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c105680 0x7f179c105a60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 -- 172.21.15.150:0/1108395575 >> 172.21.15.150:0/1108395575 conn(0x7f179c077530 msgr2=0x7f179c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 -- 172.21.15.150:0/1108395575 shutdown_connections 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.255+0000 7f17a347e640 1 -- 172.21.15.150:0/1108395575 wait complete. 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 Processor -- start 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 -- start start 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 0x7f179c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c105680 0x7f179c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f179c110ba0 con 0x7f179c100930 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f179c110a20 con 0x7f179c1012f0 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a347e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f179c110d20 con 0x7f179c105680 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a121a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a121a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38908/0 (socket says 172.21.15.150:38908) 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a121a640 1 -- 172.21.15.150:0/24418815 learned_addr learned my addr 172.21.15.150:0/24418815 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a0a19640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 0x7f179c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.256+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c105680 0x7f179c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 -- 172.21.15.150:0/24418815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c105680 msgr2=0x7f179c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c105680 0x7f179c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 -- 172.21.15.150:0/24418815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 msgr2=0x7f179c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 0x7f179c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 -- 172.21.15.150:0/24418815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c19d0d0 con 0x7f179c100930 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a0a19640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 0x7f179c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a121a640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1798005940 tx=0x7f1798004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17927fc640 1 -- 172.21.15.150:0/24418815 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798041070 con 0x7f179c100930 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17927fc640 1 -- 172.21.15.150:0/24418815 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17980054e0 con 0x7f179c100930 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c19d360 con 0x7f179c100930 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17927fc640 1 -- 172.21.15.150:0/24418815 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798003d00 con 0x7f179c100930 2024-03-18T16:06:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.257+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c1a4c50 con 0x7f179c100930 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.259+0000 7f17927fc640 1 -- 172.21.15.150:0/24418815 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f1798044070 con 0x7f179c100930 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 Processor -- start 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 -- start start 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c0d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c18c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564871b71a40 con 0x564871c0d130 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564871b1a920 con 0x564871c16820 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a6061740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564871b6dae0 con 0x564871c0da50 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17927fc640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 0x7f17880a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c0d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a0a19640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c18c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c0d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38916/0 (socket says 172.21.15.150:38916) 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a121a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a1a1b640 1 -- 172.21.15.150:0/1701505034 learned_addr learned my addr 172.21.15.150:0/1701505034 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 0x7f17880a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17927fc640 1 -- 172.21.15.150:0/24418815 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f17980f0770 con 0x7f179c100930 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a0a19640 1 -- 172.21.15.150:0/1701505034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 msgr2=0x564871c162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a0a19640 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.260+0000 7f17a0a19640 1 -- 172.21.15.150:0/1701505034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 msgr2=0x564871c0d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17a0a19640 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c0d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17a0a19640 1 -- 172.21.15.150:0/1701505034 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564871b32460 con 0x564871c16820 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17a0a19640 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c18c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f178c009fa0 tx=0x7f178c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33caa2e0a9ecaed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 0x7f17880a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1794009930 tx=0x7f1794009c00 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17877fe640 1 -- 172.21.15.150:0/1701505034 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c004030 con 0x564871c16820 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17877fe640 1 -- 172.21.15.150:0/1701505034 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f178c0041d0 con 0x564871c16820 2024-03-18T16:06:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.261+0000 7f17877fe640 1 -- 172.21.15.150:0/1701505034 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c00f7c0 con 0x564871c16820 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- 172.21.15.150:0/1701505034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 msgr2=0x564871c18c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c18c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f178c009fa0 tx=0x7f178c0118d0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- 172.21.15.150:0/1701505034 shutdown_connections 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c18c10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 --2- 172.21.15.150:0/1701505034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c0d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- 172.21.15.150:0/1701505034 >> 172.21.15.150:0/1701505034 conn(0x564871c07a50 msgr2=0x564871c07e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- 172.21.15.150:0/1701505034 shutdown_connections 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- 172.21.15.150:0/1701505034 wait complete. 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 -- start start 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 Processor -- start 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.262+0000 7f17a6061740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c13da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564871c19450 con 0x564871c0d130 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564871b2dd70 con 0x564871c16820 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x564871c195d0 con 0x564871c0da50 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 10 client.0 _ll_register_callbacks cb 0x564871885a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38922/0 (socket says 172.21.15.150:38922) 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a0a19640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c13da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:34580/0 (socket says 172.21.15.150:34580) 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 -- 172.21.15.150:0/1929318708 learned_addr learned my addr 172.21.15.150:0/1929318708 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 -- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 msgr2=0x564871c13da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c13da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 -- 172.21.15.150:0/1929318708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 msgr2=0x564871c142e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 -- 172.21.15.150:0/1929318708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564871886210 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a121a640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f17980b9980 tx=0x7f179808bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44418db746a480c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f1784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179804b040 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f1798004c20 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f178c0041d0 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17980bc910 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 10 client.15918 fetch_fsmap learned FSMap version 29 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.263+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564871952620 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f178c0041d0 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f17980c5210 con 0x564871c0d130 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 10 client.15918 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 20 client.15918 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c26990 0x564871c28dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 4 client.15918 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564871b7ef80 con 0x564871c26990 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a0a19640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c26990 0x564871c28dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f17a0a19640 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c26990 0x564871c28dc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba360d077c601f4 server_cookie=426a5c391ff11e33 in_seq=0 out_seq=0 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.264+0000 7f1784ff9640 10 client.15918 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.265+0000 7f1784ff9640 1 -- 172.21.15.150:0/1929318708 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564871b7ef80 con 0x564871c26990 2024-03-18T16:06:15.272 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.265+0000 7f1784ff9640 10 client.15918 handle_command_reply: tid=0 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.265+0000 7f17a6061740 1 client.15918 shutdown 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 msgr2=0x564871c13860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f17980b9980 tx=0x7f179808bdc0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 shutdown_connections 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c26990 0x564871c28dc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c16820 0x564871c142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c13da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f1784ff9640 0 client.15918 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 --2- 172.21.15.150:0/1929318708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c0d130 0x564871c13860 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 >> 172.21.15.150:0/1929318708 conn(0x564871c07a50 msgr2=0x564871c097a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 shutdown_connections 2024-03-18T16:06:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 1 -- 172.21.15.150:0/1929318708 wait complete. 2024-03-18T16:06:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.266+0000 7f17a6061740 20 client.15918 trim_cache size 0 max 16384 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 Processor -- start 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 -- start start 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c221f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c2e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c13820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1afc40 con 0x564871c16820 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1ad510 con 0x564871c0d130 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a6061740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1ab360 con 0x564871c0da50 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c13820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c13820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38932/0 (socket says 172.21.15.150:38932) 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a1a1b640 1 -- 172.21.15.150:0/996506694 learned_addr learned my addr 172.21.15.150:0/996506694 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a121a640 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.281+0000 7f17a0a19640 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c2e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 -- 172.21.15.150:0/996506694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 msgr2=0x564871c2e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c2e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 -- 172.21.15.150:0/996506694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 msgr2=0x564871c13820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c13820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 -- 172.21.15.150:0/996506694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564871c03310 con 0x564871c0d130 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f17a121a640 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c221f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f17980bcf40 tx=0x7f179802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fbe59da750327a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f1786ffd640 1 -- 172.21.15.150:0/996506694 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179804b050 con 0x564871c0d130 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f1786ffd640 1 -- 172.21.15.150:0/996506694 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1798005690 con 0x564871c0d130 2024-03-18T16:06:15.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.282+0000 7f1786ffd640 1 -- 172.21.15.150:0/996506694 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17980ee370 con 0x564871c0d130 2024-03-18T16:06:15.288 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- 172.21.15.150:0/996506694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 msgr2=0x564871c221f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.288 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c221f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f17980bcf40 tx=0x7f179802bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:15.288 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- 172.21.15.150:0/996506694 shutdown_connections 2024-03-18T16:06:15.288 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c13820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.288 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c2e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- 172.21.15.150:0/996506694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c221f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- 172.21.15.150:0/996506694 >> 172.21.15.150:0/996506694 conn(0x564871c07a50 msgr2=0x564871c0eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- 172.21.15.150:0/996506694 shutdown_connections 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- 172.21.15.150:0/996506694 wait complete. 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- start start 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 Processor -- start 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c1a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c1a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1ab9d0 con 0x564871c16820 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1ace40 con 0x564871c0d130 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1ac0a0 con 0x564871c0da50 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.283+0000 7f17a6061740 10 client.0 _ll_register_callbacks cb 0x7f1798009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a121a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c1a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a0a19640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c1a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38946/0 (socket says 172.21.15.150:38946) 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 -- 172.21.15.150:0/71563095 learned_addr learned my addr 172.21.15.150:0/71563095 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 -- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 msgr2=0x564871c1a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c1a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 -- 172.21.15.150:0/71563095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 msgr2=0x564871c1a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c1a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 -- 172.21.15.150:0/71563095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564871c8bf50 con 0x564871c16820 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a121a640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a1a1b640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f17940041f0 tx=0x7f1794016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3979991bd774f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f1784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794004830 con 0x564871c16820 2024-03-18T16:06:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8483+0+0 (secure 0 0 0) 0x7f1794004b20 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564871b792d0 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1794037050 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.284+0000 7f17a6061740 10 client.15924 fetch_fsmap learned FSMap version 29 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564871c8d240 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564871b792d0 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f17940053c0 con 0x564871c16820 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 10 client.15924 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 20 client.15924 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c8da10 0x564871c8fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 4 client.15924 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564871af7770 con 0x564871c8da10 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.285+0000 7f17a121a640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c8da10 0x564871c8fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.286+0000 7f1784ff9640 10 client.15924 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.286+0000 7f17a121a640 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c8da10 0x564871c8fe30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27ea97b49571e3eb server_cookie=c27be4e89df67cb0 in_seq=0 out_seq=0 2024-03-18T16:06:15.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f1784ff9640 1 -- 172.21.15.150:0/71563095 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564871af7770 con 0x564871c8da10 2024-03-18T16:06:15.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f1784ff9640 10 client.15924 handle_command_reply: tid=0 2024-03-18T16:06:15.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 client.15924 shutdown 2024-03-18T16:06:15.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 msgr2=0x564871c1ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f17940041f0 tx=0x7f1794016000 comp rx=0 tx=0).stop 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 shutdown_connections 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x564871c8da10 0x564871c8fe30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564871c16820 0x564871c1ade0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564871c0da50 0x564871c1a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 --2- 172.21.15.150:0/71563095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564871c0d130 0x564871c1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f1784ff9640 0 client.15924 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 >> 172.21.15.150:0/71563095 conn(0x564871c07a50 msgr2=0x564871c06320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 shutdown_connections 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 1 -- 172.21.15.150:0/71563095 wait complete. 2024-03-18T16:06:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.295+0000 7f17a6061740 20 client.15924 trim_cache size 0 max 16384 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 msgr2=0x7f17880a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 0x7f17880a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1794009930 tx=0x7f1794009c00 comp rx=0 tx=0).stop 2024-03-18T16:06:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 msgr2=0x7f179c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1798005940 tx=0x7f1798004270 comp rx=0 tx=0).stop 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 shutdown_connections 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f17880a2a10 0x7f17880a4ed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f179c105680 0x7f179c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f179c1012f0 0x7f179c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 --2- 172.21.15.150:0/24418815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f179c100930 0x7f179c1980e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 >> 172.21.15.150:0/24418815 conn(0x7f179c077530 msgr2=0x7f179c069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 shutdown_connections 2024-03-18T16:06:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:15.296+0000 7f17a347e640 1 -- 172.21.15.150:0/24418815 wait complete. 2024-03-18T16:06:15.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 614679 11.35 MB/sec execute 180 sec latency 37.332 ms 2024-03-18T16:06:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:15 smithi191 ceph-mon[36784]: pgmap v189: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:15 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:15 smithi156 ceph-mon[35919]: pgmap v189: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:15 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:15 smithi150 ceph-mon[31946]: pgmap v189: 97 pgs: 97 active+clean; 376 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-18T16:06:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:15 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:16.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 616129 11.32 MB/sec execute 181 sec latency 21.083 ms 2024-03-18T16:06:16.482 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-18T16:06:16.483 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 systemd[1]: Stopped Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:06:16.483 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service: Consumed 1.476s CPU time. 2024-03-18T16:06:16.483 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 systemd[1]: Starting Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:06:16.483 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 podman[72130]: 2024-03-18 16:06:16.366345451 +0000 UTC m=+0.024756166 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:16.483 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 podman[72130]: 2024-03-18 16:06:16.481673931 +0000 UTC m=+0.140084619 container create 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T16:06:16.745 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:16 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:16.746 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:16 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:16.746 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:16 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:06:17.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:16 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:17.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:16 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:17.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:16 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:06:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:16 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:16 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:16 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 podman[72130]: 2024-03-18 16:06:16.744881104 +0000 UTC m=+0.403291808 container init 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 podman[72130]: 2024-03-18 16:06:16.748825088 +0000 UTC m=+0.407235793 container start 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 ceph-mds[72148]: -- 172.21.15.191:0/2417479719 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586aa0c9a40 con 0x5586aa09d000 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[72144]: starting mds.f at 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 bash[72130]: 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 2024-03-18T16:06:17.211 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:06:16 smithi191 systemd[1]: Started Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:06:17.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 618284 11.33 MB/sec execute 182 sec latency 19.250 ms 2024-03-18T16:06:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:17 smithi191 ceph-mon[36784]: pgmap v190: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 302 op/s 2024-03-18T16:06:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:17 smithi191 ceph-mon[36784]: daemon mds.f restarted 2024-03-18T16:06:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:17 smithi191 ceph-mon[36784]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:06:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:17 smithi156 ceph-mon[35919]: pgmap v190: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 302 op/s 2024-03-18T16:06:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:17 smithi156 ceph-mon[35919]: daemon mds.f restarted 2024-03-18T16:06:18.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:17 smithi156 ceph-mon[35919]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:06:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:17 smithi150 ceph-mon[31946]: pgmap v190: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 302 op/s 2024-03-18T16:06:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:17 smithi150 ceph-mon[31946]: daemon mds.f restarted 2024-03-18T16:06:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:17 smithi150 ceph-mon[31946]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:06:18.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 619687 11.30 MB/sec execute 183 sec latency 23.438 ms 2024-03-18T16:06:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:18 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6834/851431923,v1:172.21.15.191:6835/851431923] up:boot 2024-03-18T16:06:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:18 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T16:06:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:18 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:18 smithi191 ceph-mon[36784]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:06:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:18 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6834/851431923,v1:172.21.15.191:6835/851431923] up:boot 2024-03-18T16:06:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:18 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T16:06:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:18 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:19.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:18 smithi156 ceph-mon[35919]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:06:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:18 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6834/851431923,v1:172.21.15.191:6835/851431923] up:boot 2024-03-18T16:06:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:18 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-18T16:06:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:18 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:18 smithi150 ceph-mon[31946]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T16:06:19.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 621771 11.31 MB/sec execute 184 sec latency 129.082 ms 2024-03-18T16:06:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:19 smithi191 ceph-mon[36784]: pgmap v192: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 228 op/s 2024-03-18T16:06:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:19 smithi156 ceph-mon[35919]: pgmap v192: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 228 op/s 2024-03-18T16:06:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:19 smithi150 ceph-mon[31946]: pgmap v192: 97 pgs: 97 active+clean; 383 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 228 op/s 2024-03-18T16:06:20.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 623136 11.28 MB/sec execute 185 sec latency 22.967 ms 2024-03-18T16:06:21.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 625375 11.29 MB/sec execute 186 sec latency 17.406 ms 2024-03-18T16:06:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:21 smithi191 ceph-mon[36784]: pgmap v193: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:21 smithi156 ceph-mon[35919]: pgmap v193: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:21 smithi150 ceph-mon[31946]: pgmap v193: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:22.327 INFO:tasks.workunit.client.0.smithi150.stdout: 1 626938 11.26 MB/sec execute 187 sec latency 20.627 ms 2024-03-18T16:06:23.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 629164 11.27 MB/sec execute 188 sec latency 13.332 ms 2024-03-18T16:06:23.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:23 smithi191 ceph-mon[36784]: pgmap v194: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:23 smithi156 ceph-mon[35919]: pgmap v194: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:23 smithi150 ceph-mon[31946]: pgmap v194: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:24.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 630758 11.24 MB/sec execute 189 sec latency 77.217 ms 2024-03-18T16:06:25.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 632721 11.26 MB/sec execute 190 sec latency 35.222 ms 2024-03-18T16:06:25.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:25 smithi191 ceph-mon[36784]: pgmap v195: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:25 smithi156 ceph-mon[35919]: pgmap v195: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:25 smithi150 ceph-mon[31946]: pgmap v195: 97 pgs: 97 active+clean; 389 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 226 op/s 2024-03-18T16:06:26.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 634467 11.23 MB/sec execute 191 sec latency 32.492 ms 2024-03-18T16:06:27.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 636304 11.24 MB/sec execute 192 sec latency 19.988 ms 2024-03-18T16:06:27.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:27 smithi191 ceph-mon[36784]: pgmap v196: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-18T16:06:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:27 smithi156 ceph-mon[35919]: pgmap v196: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-18T16:06:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:27 smithi150 ceph-mon[31946]: pgmap v196: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-18T16:06:28.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 638208 11.22 MB/sec execute 193 sec latency 21.020 ms 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 -- 172.21.15.150:0/3405619065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24541012d0 msgr2=0x7f245410d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 --2- 172.21.15.150:0/3405619065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24541012d0 0x7f245410d500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f244c009ca0 tx=0x7f244c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 -- 172.21.15.150:0/3405619065 shutdown_connections 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 --2- 172.21.15.150:0/3405619065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24541012d0 0x7f245410d500 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 --2- 172.21.15.150:0/3405619065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 0x7f2454100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 --2- 172.21.15.150:0/3405619065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2454106760 0x7f2454106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 -- 172.21.15.150:0/3405619065 >> 172.21.15.150:0/3405619065 conn(0x7f2454077540 msgr2=0x7f2454077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 -- 172.21.15.150:0/3405619065 shutdown_connections 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.392+0000 7f245ac12640 1 -- 172.21.15.150:0/3405619065 wait complete. 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 Processor -- start 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 -- start start 2024-03-18T16:06:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 0x7f24541023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24541012d0 0x7f24541028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2454110ae0 con 0x7f2454106760 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2454110960 con 0x7f24541012d0 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f245ac12640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2454110c60 con 0x7f2454100930 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24589ae640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 0x7f24541023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44444/0 (socket says 172.21.15.150:44444) 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 -- 172.21.15.150:0/1847805570 learned_addr learned my addr 172.21.15.150:0/1847805570 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f2453fff640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24541012d0 0x7f24541028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 -- 172.21.15.150:0/1847805570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 msgr2=0x7f24541023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 0x7f24541023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 -- 172.21.15.150:0/1847805570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24541012d0 msgr2=0x7f24541028e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24541012d0 0x7f24541028e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 -- 172.21.15.150:0/1847805570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24541a6470 con 0x7f2454106760 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.393+0000 7f24591af640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f244c02c700 tx=0x7f244c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.394+0000 7f2451ffb640 1 -- 172.21.15.150:0/1847805570 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c041020 con 0x7f2454106760 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.394+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24541a6700 con 0x7f2454106760 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.394+0000 7f2451ffb640 1 -- 172.21.15.150:0/1847805570 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c034cd0 con 0x7f2454106760 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.394+0000 7f2451ffb640 1 -- 172.21.15.150:0/1847805570 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c039040 con 0x7f2454106760 2024-03-18T16:06:28.398 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.394+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24541a6c40 con 0x7f2454106760 2024-03-18T16:06:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.396+0000 7f2451ffb640 1 -- 172.21.15.150:0/1847805570 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f244c044020 con 0x7f2454106760 2024-03-18T16:06:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 Processor -- start 2024-03-18T16:06:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 -- start start 2024-03-18T16:06:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2451ffb640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 0x7f24440a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d7881b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d7881dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558d78776a40 con 0x558d78812130 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558d7871f920 con 0x558d7881b820 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f245d800740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558d78772ae0 con 0x558d78812a50 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2451ffb640 1 -- 172.21.15.150:0/1847805570 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f244c0f0900 con 0x7f2454106760 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2453fff640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 0x7f24440a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44454/0 (socket says 172.21.15.150:44454) 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 -- 172.21.15.150:0/2332975700 learned_addr learned my addr 172.21.15.150:0/2332975700 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2453fff640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d7881b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24591af640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d7881dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.405 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 -- 172.21.15.150:0/2332975700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 msgr2=0x558d7881b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d7881b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 -- 172.21.15.150:0/2332975700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 msgr2=0x558d7881dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d7881dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 -- 172.21.15.150:0/2332975700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d78737460 con 0x558d78812130 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2453fff640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d7881b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f24589ae640 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2448009c70 tx=0x7f2448011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e3fedd3f54c5b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.397+0000 7f2453fff640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 0x7f24440a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558d78811be0 tx=0x7f2440009840 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f243effd640 1 -- 172.21.15.150:0/2332975700 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448004230 con 0x558d78812130 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f243effd640 1 -- 172.21.15.150:0/2332975700 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24480043d0 con 0x558d78812130 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f243effd640 1 -- 172.21.15.150:0/2332975700 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448019800 con 0x558d78812130 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 -- 172.21.15.150:0/2332975700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 msgr2=0x558d78812510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2448009c70 tx=0x7f2448011810 comp rx=0 tx=0).stop 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 -- 172.21.15.150:0/2332975700 shutdown_connections 2024-03-18T16:06:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d7881dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d7881b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 --2- 172.21.15.150:0/2332975700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78812510 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 -- 172.21.15.150:0/2332975700 >> 172.21.15.150:0/2332975700 conn(0x558d7880ca50 msgr2=0x558d7880ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 -- 172.21.15.150:0/2332975700 shutdown_connections 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.398+0000 7f245d800740 1 -- 172.21.15.150:0/2332975700 wait complete. 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 -- start start 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 Processor -- start 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78818d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d78819290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x558d7881e450 con 0x558d78812130 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558d78732d70 con 0x558d7881b820 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x558d7881e5d0 con 0x558d78812a50 2024-03-18T16:06:28.407 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f245d800740 10 client.0 _ll_register_callbacks cb 0x558d7848aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f24589ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f24589ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44470/0 (socket says 172.21.15.150:44470) 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f24589ae640 1 -- 172.21.15.150:0/2491133946 learned_addr learned my addr 172.21.15.150:0/2491133946 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f2453fff640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78818d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.399+0000 7f24591af640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d78819290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 -- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 msgr2=0x558d78818d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78818d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 -- 172.21.15.150:0/2491133946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 msgr2=0x558d78819290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d78819290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 -- 172.21.15.150:0/2491133946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d7848b210 con 0x558d78812130 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f24589ae640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2448000c00 tx=0x7f24480056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943a0eed0e6dfb18 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448027070 con 0x558d78812130 2024-03-18T16:06:28.408 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f2448003800 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24480043d0 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2448033050 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 10 client.15942 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558d78557620 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24480043d0 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2448018070 con 0x558d78812130 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 10 client.15942 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 20 client.15942 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d7882b840 0x558d7882dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 4 client.15942 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.400+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d78783f80 con 0x558d7882b840 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.401+0000 7f2453fff640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d7882b840 0x558d7882dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.401+0000 7f2433fff640 10 client.15942 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:28.409 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.401+0000 7f2453fff640 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d7882b840 0x558d7882dc30 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb36e105b305eaf6 server_cookie=c001c176e684e531 in_seq=0 out_seq=0 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f2433fff640 1 -- 172.21.15.150:0/2491133946 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558d78783f80 con 0x558d7882b840 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f2433fff640 10 client.15942 handle_command_reply: tid=0 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 client.15942 shutdown 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 msgr2=0x558d78818810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2448000c00 tx=0x7f24480056d0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 shutdown_connections 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d7882b840 0x558d7882dc30 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d7881b820 0x558d78819290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78818d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f2433fff640 0 client.15942 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 --2- 172.21.15.150:0/2491133946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812130 0x558d78818810 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 >> 172.21.15.150:0/2491133946 conn(0x558d7880ca50 msgr2=0x558d7880e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 shutdown_connections 2024-03-18T16:06:28.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 1 -- 172.21.15.150:0/2491133946 wait complete. 2024-03-18T16:06:28.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.402+0000 7f245d800740 20 client.15942 trim_cache size 0 max 16384 2024-03-18T16:06:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 Processor -- start 2024-03-18T16:06:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 -- start start 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24541b1e10 con 0x558d7881b820 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24541af6e0 con 0x558d78812130 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f245d800740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f24541ad530 con 0x558d78812a50 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f24591af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f24589ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f2453fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f24591af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44478/0 (socket says 172.21.15.150:44478) 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f24589ae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:45442/0 (socket says 172.21.15.150:45442) 2024-03-18T16:06:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f2453fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:40112/0 (socket says 172.21.15.150:40112) 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.418+0000 7f24591af640 1 -- 172.21.15.150:0/1231917591 learned_addr learned my addr 172.21.15.150:0/1231917591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 -- 172.21.15.150:0/1231917591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 msgr2=0x558d78807a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 -- 172.21.15.150:0/1231917591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 msgr2=0x558d788271f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 -- 172.21.15.150:0/1231917591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d788177e0 con 0x558d7881b820 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24589ae640 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f2453fff640 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f24591af640 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f244c0b9b10 tx=0x7f244c0dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2eca4af70b08004 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f243e7fc640 1 -- 172.21.15.150:0/1231917591 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0bcaa0 con 0x558d7881b820 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f243e7fc640 1 -- 172.21.15.150:0/1231917591 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c034920 con 0x558d7881b820 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.419+0000 7f243e7fc640 1 -- 172.21.15.150:0/1231917591 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0b97b0 con 0x558d7881b820 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- 172.21.15.150:0/1231917591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 msgr2=0x558d78833480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f244c0b9b10 tx=0x7f244c0dd040 comp rx=0 tx=0).stop 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- 172.21.15.150:0/1231917591 shutdown_connections 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d7881b820 0x558d78833480 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d78812a50 0x558d78807a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- 172.21.15.150:0/1231917591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d788271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- 172.21.15.150:0/1231917591 >> 172.21.15.150:0/1231917591 conn(0x558d7880ca50 msgr2=0x558d7880e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- 172.21.15.150:0/1231917591 shutdown_connections 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- 172.21.15.150:0/1231917591 wait complete. 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- start start 2024-03-18T16:06:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 Processor -- start 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d78823a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d7881b820 0x558d78820d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24541adba0 con 0x558d78812a50 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24541af010 con 0x558d78812130 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.420+0000 7f245d800740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f24541ae270 con 0x558d7881b820 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f245d800740 10 client.0 _ll_register_callbacks cb 0x7f244c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44490/0 (socket says 172.21.15.150:44490) 2024-03-18T16:06:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 -- 172.21.15.150:0/998228292 learned_addr learned my addr 172.21.15.150:0/998228292 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f24589ae640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d78823a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f24591af640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d7881b820 0x558d78820d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 -- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d7881b820 msgr2=0x558d78820d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d7881b820 0x558d78820d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 -- 172.21.15.150:0/998228292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 msgr2=0x558d78823a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d78823a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 -- 172.21.15.150:0/998228292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d78821370 con 0x558d78812a50 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f24589ae640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d78823a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2453fff640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2440009710 tx=0x7f2440005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a584a1eab745b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2433fff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244002aa30 con 0x558d78812a50 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f244002abd0 con 0x558d78812a50 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f244c034920 con 0x558d78812a50 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2440037050 con 0x558d78812a50 2024-03-18T16:06:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f245d800740 10 client.15948 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.421+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558d78892900 con 0x558d78812a50 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f244c034920 con 0x558d78812a50 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f244001f040 con 0x558d78812a50 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 10 client.15948 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 10 client.15948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 20 client.15948 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d788930b0 0x558d788954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 4 client.15948 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d7877e2d0 con 0x558d788930b0 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f24589ae640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d788930b0 0x558d788954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f2433fff640 10 client.15948 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.422+0000 7f24589ae640 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d788930b0 0x558d788954d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ea2f6883f79584 server_cookie=baa3c80742b6754e in_seq=0 out_seq=0 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f2433fff640 1 -- 172.21.15.150:0/998228292 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d7877e2d0 con 0x558d788930b0 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f2433fff640 10 client.15948 handle_command_reply: tid=0 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 client.15948 shutdown 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 msgr2=0x558d788207f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2440009710 tx=0x7f2440005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 shutdown_connections 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x558d788930b0 0x558d788954d0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x558d7881b820 0x558d78820d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d78812a50 0x558d788207f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 --2- 172.21.15.150:0/998228292 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558d78812130 0x558d78823a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 >> 172.21.15.150:0/998228292 conn(0x558d7880ca50 msgr2=0x558d788187b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f2433fff640 0 client.15948 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 shutdown_connections 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 1 -- 172.21.15.150:0/998228292 wait complete. 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.432+0000 7f245d800740 20 client.15948 trim_cache size 0 max 16384 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:06:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:06:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 msgr2=0x7f24440a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 0x7f24440a4ed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558d78811be0 tx=0x7f2440009840 comp rx=0 tx=0).stop 2024-03-18T16:06:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 msgr2=0x7f2454102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f244c02c700 tx=0x7f244c032f20 comp rx=0 tx=0).stop 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 shutdown_connections 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f24440a2a10 0x7f24440a4ed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2454106760 0x7f2454102e20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24541012d0 0x7f24541028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 --2- 172.21.15.150:0/1847805570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2454100930 0x7f24541023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 >> 172.21.15.150:0/1847805570 conn(0x7f2454077540 msgr2=0x7f24540ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 shutdown_connections 2024-03-18T16:06:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:28.433+0000 7f245ac12640 1 -- 172.21.15.150:0/1847805570 wait complete. 2024-03-18T16:06:29.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 639850 11.22 MB/sec execute 194 sec latency 83.843 ms 2024-03-18T16:06:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:29 smithi191 ceph-mon[36784]: pgmap v197: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-18T16:06:30.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:29 smithi156 ceph-mon[35919]: pgmap v197: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-18T16:06:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:29 smithi150 ceph-mon[31946]: pgmap v197: 97 pgs: 97 active+clean; 397 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 201 op/s 2024-03-18T16:06:30.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 641082 11.19 MB/sec execute 195 sec latency 34.263 ms 2024-03-18T16:06:31.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 643261 11.20 MB/sec execute 196 sec latency 16.846 ms 2024-03-18T16:06:31.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:31 smithi191 ceph-mon[36784]: pgmap v198: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-03-18T16:06:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:31 smithi156 ceph-mon[35919]: pgmap v198: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-03-18T16:06:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:31 smithi150 ceph-mon[31946]: pgmap v198: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 286 op/s 2024-03-18T16:06:32.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 644449 11.17 MB/sec execute 197 sec latency 20.003 ms 2024-03-18T16:06:33.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 646187 11.16 MB/sec execute 198 sec latency 21.729 ms 2024-03-18T16:06:33.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:33 smithi191 ceph-mon[36784]: pgmap v199: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:33 smithi156 ceph-mon[35919]: pgmap v199: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:33 smithi150 ceph-mon[31946]: pgmap v199: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:34.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 647741 11.16 MB/sec execute 199 sec latency 61.039 ms 2024-03-18T16:06:35.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 649857 11.16 MB/sec execute 200 sec latency 33.765 ms 2024-03-18T16:06:35.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:35 smithi191 ceph-mon[36784]: pgmap v200: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:35 smithi156 ceph-mon[35919]: pgmap v200: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:35 smithi150 ceph-mon[31946]: pgmap v200: 97 pgs: 97 active+clean; 404 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-03-18T16:06:36.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 651152 11.13 MB/sec execute 201 sec latency 136.766 ms 2024-03-18T16:06:37.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 653242 11.13 MB/sec execute 202 sec latency 53.350 ms 2024-03-18T16:06:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:37 smithi191 ceph-mon[36784]: pgmap v201: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:37 smithi156 ceph-mon[35919]: pgmap v201: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:38.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:37 smithi150 ceph-mon[31946]: pgmap v201: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:38.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 654893 11.12 MB/sec execute 203 sec latency 19.780 ms 2024-03-18T16:06:39.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 656990 11.13 MB/sec execute 204 sec latency 62.171 ms 2024-03-18T16:06:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:39 smithi191 ceph-mon[36784]: pgmap v202: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-18T16:06:40.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:39 smithi156 ceph-mon[35919]: pgmap v202: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-18T16:06:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:39 smithi150 ceph-mon[31946]: pgmap v202: 97 pgs: 97 active+clean; 411 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-18T16:06:40.328 INFO:tasks.workunit.client.0.smithi150.stdout: 1 658513 11.11 MB/sec execute 205 sec latency 19.240 ms 2024-03-18T16:06:41.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 660776 11.12 MB/sec execute 206 sec latency 17.445 ms 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- 172.21.15.150:0/1908377104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 msgr2=0x7fa9540692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/1908377104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa9540692c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa948009930 tx=0x7fa94802bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- 172.21.15.150:0/1908377104 shutdown_connections 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/1908377104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa95410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/1908377104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa9540692c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/1908377104 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 0x7fa954107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- 172.21.15.150:0/1908377104 >> 172.21.15.150:0/1908377104 conn(0x7fa954077530 msgr2=0x7fa954077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- 172.21.15.150:0/1908377104 shutdown_connections 2024-03-18T16:06:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- 172.21.15.150:0/1908377104 wait complete. 2024-03-18T16:06:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 Processor -- start 2024-03-18T16:06:41.525 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.521+0000 7fa95a9bc640 1 -- start start 2024-03-18T16:06:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa954100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 0x7fa954101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa954111ba0 con 0x7fa954069800 2024-03-18T16:06:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa954111a20 con 0x7fa954068e60 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa95a9bc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa954111d20 con 0x7fa9541078b0 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40548/0 (socket says 172.21.15.150:40548) 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 -- 172.21.15.150:0/581384968 learned_addr learned my addr 172.21.15.150:0/581384968 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa953fff640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa954100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa958f59640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 0x7fa954101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 -- 172.21.15.150:0/581384968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 msgr2=0x7fa954101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 0x7fa954101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 -- 172.21.15.150:0/581384968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 msgr2=0x7fa954100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa954100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 -- 172.21.15.150:0/581384968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9541a6500 con 0x7fa954069800 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9537fe640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa948009230 tx=0x7fa948004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.522+0000 7fa9517fa640 1 -- 172.21.15.150:0/581384968 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948041070 con 0x7fa954069800 2024-03-18T16:06:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.523+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9541a6790 con 0x7fa954069800 2024-03-18T16:06:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.523+0000 7fa9517fa640 1 -- 172.21.15.150:0/581384968 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9480054e0 con 0x7fa954069800 2024-03-18T16:06:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.523+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9541a6c80 con 0x7fa954069800 2024-03-18T16:06:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.523+0000 7fa9517fa640 1 -- 172.21.15.150:0/581384968 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94804b060 con 0x7fa954069800 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.525+0000 7fa9517fa640 1 -- 172.21.15.150:0/581384968 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fa948044070 con 0x7fa954069800 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.525+0000 7fa95d59b740 1 Processor -- start 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.525+0000 7fa95d59b740 1 -- start start 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.525+0000 7fa9517fa640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 0x7fa93c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.525+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751e3820 0x55e5751e5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa95d59b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e57513ea40 con 0x55e5751daa50 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa958f59640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 0x7fa93c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa95d59b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e5750e7920 con 0x55e5751da130 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa95d59b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e57513aae0 con 0x55e5751e3820 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9517fa640 1 -- 172.21.15.150:0/581384968 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7fa9480f0c90 con 0x7fa954069800 2024-03-18T16:06:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:37220/0 (socket says 172.21.15.150:37220) 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa953fff640 1 -- 172.21.15.150:0/1277061804 learned_addr learned my addr 172.21.15.150:0/1277061804 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa958f59640 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751e3820 0x55e5751e5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 -- 172.21.15.150:0/1277061804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751e3820 msgr2=0x55e5751e5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751e3820 0x55e5751e5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 -- 172.21.15.150:0/1277061804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 msgr2=0x55e5751da510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 -- 172.21.15.150:0/1277061804 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5750ff460 con 0x55e5751daa50 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa953fff640 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa958f59640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 0x7fa93c0a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009fd0 tx=0x7fa94c011bc0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa9537fe640 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e32e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa9480ee510 tx=0x7fa9480058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7531ed7270cc3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa93a7fc640 1 -- 172.21.15.150:0/1277061804 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480bce30 con 0x55e5751daa50 2024-03-18T16:06:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa93a7fc640 1 -- 172.21.15.150:0/1277061804 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa948004c80 con 0x55e5751daa50 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.526+0000 7fa93a7fc640 1 -- 172.21.15.150:0/1277061804 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480ee680 con 0x55e5751daa50 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- 172.21.15.150:0/1277061804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 msgr2=0x55e5751e32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e32e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa9480ee510 tx=0x7fa9480058b0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- 172.21.15.150:0/1277061804 shutdown_connections 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751e3820 0x55e5751e5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e32e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1277061804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751da510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- 172.21.15.150:0/1277061804 >> 172.21.15.150:0/1277061804 conn(0x55e5751d4a50 msgr2=0x55e5751d4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- 172.21.15.150:0/1277061804 shutdown_connections 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- 172.21.15.150:0/1277061804 wait complete. 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.527+0000 7fa95d59b740 1 -- start start 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 Processor -- start 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751e0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751e1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e5751e6210 con 0x55e5751daa50 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e5750fad70 con 0x55e5751e3820 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e5751e6390 con 0x55e5751da130 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa95d59b740 10 client.0 _ll_register_callbacks cb 0x55e574e52a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa953fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40556/0 (socket says 172.21.15.150:40556) 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 -- 172.21.15.150:0/635474489 learned_addr learned my addr 172.21.15.150:0/635474489 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa958f59640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 -- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 msgr2=0x55e5751e0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751e0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 -- 172.21.15.150:0/635474489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 msgr2=0x55e5751e1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751e1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 -- 172.21.15.150:0/635474489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e574e53210 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9537fe640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa9480ee570 tx=0x7fa948003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf111bc07c285852 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.528+0000 7fa9337fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480d5070 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7fa9480d6260 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa948004c80 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa94802ccd0 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 10 client.15960 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e574f1f620 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa948004c80 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa94809d030 con 0x55e5751daa50 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 10 client.15960 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 20 client.15960 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5751f3570 0x55e5751f5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 4 client.15960 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e57514bf80 con 0x55e5751f3570 2024-03-18T16:06:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.529+0000 7fa953fff640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5751f3570 0x55e5751f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.530+0000 7fa9337fe640 10 client.15960 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.530+0000 7fa953fff640 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5751f3570 0x55e5751f5960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b7370e40256b3d server_cookie=9c7b73f4e8084fe0 in_seq=0 out_seq=0 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa9337fe640 1 -- 172.21.15.150:0/635474489 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e57514bf80 con 0x55e5751f3570 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa9337fe640 10 client.15960 handle_command_reply: tid=0 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 client.15960 shutdown 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 msgr2=0x55e5751e0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa9480ee570 tx=0x7fa948003760 comp rx=0 tx=0).stop 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 shutdown_connections 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5751f3570 0x55e5751f5960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751e1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751e0dd0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa9337fe640 0 client.15960 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 --2- 172.21.15.150:0/635474489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 >> 172.21.15.150:0/635474489 conn(0x55e5751d4a50 msgr2=0x55e5751d67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 shutdown_connections 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 1 -- 172.21.15.150:0/635474489 wait complete. 2024-03-18T16:06:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.531+0000 7fa95d59b740 20 client.15960 trim_cache size 0 max 16384 2024-03-18T16:06:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 Processor -- start 2024-03-18T16:06:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 -- start start 2024-03-18T16:06:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751eef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751fb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541b1e70 con 0x55e5751daa50 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541af740 con 0x55e5751e3820 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa95d59b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541ad590 con 0x55e5751da130 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa953fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40562/0 (socket says 172.21.15.150:40562) 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa953fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:53458/0 (socket says 172.21.15.150:53458) 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa9537fe640 1 -- 172.21.15.150:0/921473623 learned_addr learned my addr 172.21.15.150:0/921473623 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa958f59640 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751fb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.546+0000 7fa9537fe640 1 -- 172.21.15.150:0/921473623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 msgr2=0x55e5751eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa9537fe640 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa9537fe640 1 -- 172.21.15.150:0/921473623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 msgr2=0x55e5751fb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa9537fe640 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751fb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa9537fe640 1 -- 172.21.15.150:0/921473623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5751f5ef0 con 0x55e5751daa50 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa9537fe640 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa9480b9930 tx=0x7fa94802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e573fff12f840c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa939ffb640 1 -- 172.21.15.150:0/921473623 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480d5070 con 0x55e5751daa50 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa939ffb640 1 -- 172.21.15.150:0/921473623 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa948004c80 con 0x55e5751daa50 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa939ffb640 1 -- 172.21.15.150:0/921473623 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480d6260 con 0x55e5751daa50 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa95d59b740 1 -- 172.21.15.150:0/921473623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 msgr2=0x55e5751d65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.547+0000 7fa95d59b740 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa9480b9930 tx=0x7fa94802bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- 172.21.15.150:0/921473623 shutdown_connections 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751e3820 0x55e5751fb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751daa50 0x55e5751d65b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- 172.21.15.150:0/921473623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751da130 0x55e5751eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- 172.21.15.150:0/921473623 >> 172.21.15.150:0/921473623 conn(0x55e5751d4a50 msgr2=0x55e5751d5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- 172.21.15.150:0/921473623 shutdown_connections 2024-03-18T16:06:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- 172.21.15.150:0/921473623 wait complete. 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- start start 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 Processor -- start 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751daa50 0x55e5751e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541adc00 con 0x55e5751e3820 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541af070 con 0x55e5751da130 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.548+0000 7fa95d59b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541ae2d0 con 0x55e5751daa50 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa95d59b740 10 client.0 _ll_register_callbacks cb 0x7fa944009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa9537fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751daa50 0x55e5751e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40574/0 (socket says 172.21.15.150:40574) 2024-03-18T16:06:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 -- 172.21.15.150:0/1176646308 learned_addr learned my addr 172.21.15.150:0/1176646308 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa953fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:37250/0 (socket says 172.21.15.150:37250) 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 -- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751daa50 msgr2=0x55e5751e7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751daa50 0x55e5751e7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 -- 172.21.15.150:0/1176646308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 msgr2=0x55e5751e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 -- 172.21.15.150:0/1176646308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e575258cf0 con 0x55e5751e3820 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa953fff640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa958f59640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005fd0 tx=0x7fa94c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eece27608842a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa9337fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c02f060 con 0x55e5751e3820 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7fa94c004720 con 0x55e5751e3820 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa948004c80 con 0x55e5751e3820 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa94c01a040 con 0x55e5751e3820 2024-03-18T16:06:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.549+0000 7fa95d59b740 10 client.15966 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e57525a000 con 0x55e5751e3820 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa948004c80 con 0x55e5751e3820 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa94c01d030 con 0x55e5751e3820 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 10 client.15966 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 20 client.15966 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e57525a7d0 0x55e57525cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 4 client.15966 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5751462d0 con 0x55e57525a7d0 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa953fff640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e57525a7d0 0x55e57525cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa953fff640 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e57525a7d0 0x55e57525cbf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a07787f78d3c85c server_cookie=6370e66b848d4bd2 in_seq=0 out_seq=0 2024-03-18T16:06:41.558 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.550+0000 7fa9337fe640 10 client.15966 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa9337fe640 1 -- 172.21.15.150:0/1176646308 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5751462d0 con 0x55e57525a7d0 2024-03-18T16:06:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa9337fe640 10 client.15966 handle_command_reply: tid=0 2024-03-18T16:06:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 client.15966 shutdown 2024-03-18T16:06:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 msgr2=0x55e5752587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005fd0 tx=0x7fa94c004ab0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 shutdown_connections 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e57525a7d0 0x55e57525cbf0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5751e3820 0x55e5752587b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5751daa50 0x55e5751e7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa9337fe640 0 client.15966 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 --2- 172.21.15.150:0/1176646308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5751da130 0x55e5751e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.560+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 >> 172.21.15.150:0/1176646308 conn(0x55e5751d4a50 msgr2=0x55e5751d6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 shutdown_connections 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95d59b740 1 -- 172.21.15.150:0/1176646308 wait complete. 2024-03-18T16:06:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95d59b740 20 client.15966 trim_cache size 0 max 16384 2024-03-18T16:06:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:06:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:06:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:06:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:06:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 msgr2=0x7fa93c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 0x7fa93c0a4ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009fd0 tx=0x7fa94c011bc0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 msgr2=0x7fa954100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa948009230 tx=0x7fa948004270 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 shutdown_connections 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa93c0a2a10 0x7fa93c0a4ed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9541078b0 0x7fa954101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa954069800 0x7fa954100f40 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 --2- 172.21.15.150:0/581384968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa954068e60 0x7fa954100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 >> 172.21.15.150:0/581384968 conn(0x7fa954077530 msgr2=0x7fa9540fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 shutdown_connections 2024-03-18T16:06:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:41.561+0000 7fa95a9bc640 1 -- 172.21.15.150:0/581384968 wait complete. 2024-03-18T16:06:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:41 smithi191 ceph-mon[36784]: pgmap v203: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-03-18T16:06:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:41 smithi156 ceph-mon[35919]: pgmap v203: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-03-18T16:06:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:41 smithi150 ceph-mon[31946]: pgmap v203: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 268 op/s 2024-03-18T16:06:42.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 662198 11.10 MB/sec execute 207 sec latency 41.098 ms 2024-03-18T16:06:43.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 664448 11.11 MB/sec execute 208 sec latency 30.869 ms 2024-03-18T16:06:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:43 smithi191 ceph-mon[36784]: pgmap v204: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:43 smithi156 ceph-mon[35919]: pgmap v204: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:43 smithi150 ceph-mon[31946]: pgmap v204: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:44.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 665765 11.08 MB/sec execute 209 sec latency 105.825 ms 2024-03-18T16:06:45.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 667926 11.09 MB/sec execute 210 sec latency 45.504 ms 2024-03-18T16:06:45.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:45 smithi191 ceph-mon[36784]: pgmap v205: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:45 smithi156 ceph-mon[35919]: pgmap v205: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:45 smithi150 ceph-mon[31946]: pgmap v205: 97 pgs: 97 active+clean; 417 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 174 op/s 2024-03-18T16:06:46.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 669517 11.07 MB/sec execute 211 sec latency 20.755 ms 2024-03-18T16:06:47.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 671582 11.08 MB/sec execute 212 sec latency 62.405 ms 2024-03-18T16:06:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:47 smithi191 ceph-mon[36784]: pgmap v206: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-03-18T16:06:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:47 smithi156 ceph-mon[35919]: pgmap v206: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-03-18T16:06:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:47 smithi150 ceph-mon[31946]: pgmap v206: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 269 op/s 2024-03-18T16:06:48.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 673565 11.06 MB/sec execute 213 sec latency 21.730 ms 2024-03-18T16:06:49.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 675230 11.07 MB/sec execute 214 sec latency 73.848 ms 2024-03-18T16:06:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:49 smithi191 ceph-mon[36784]: pgmap v207: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-03-18T16:06:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:49 smithi156 ceph-mon[35919]: pgmap v207: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-03-18T16:06:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:49 smithi150 ceph-mon[31946]: pgmap v207: 97 pgs: 97 active+clean; 424 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 185 op/s 2024-03-18T16:06:50.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 676966 11.04 MB/sec execute 215 sec latency 23.770 ms 2024-03-18T16:06:51.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 678903 11.05 MB/sec execute 216 sec latency 12.814 ms 2024-03-18T16:06:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:51 smithi191 ceph-mon[36784]: pgmap v208: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:51 smithi156 ceph-mon[35919]: pgmap v208: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:51 smithi150 ceph-mon[31946]: pgmap v208: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 275 op/s 2024-03-18T16:06:52.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 680739 11.03 MB/sec execute 217 sec latency 24.263 ms 2024-03-18T16:06:53.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 682524 11.04 MB/sec execute 218 sec latency 24.502 ms 2024-03-18T16:06:53.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:53 smithi191 ceph-mon[36784]: pgmap v209: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:53 smithi156 ceph-mon[35919]: pgmap v209: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:53 smithi150 ceph-mon[31946]: pgmap v209: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:54.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 684265 11.02 MB/sec execute 219 sec latency 63.085 ms 2024-03-18T16:06:54.659 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 -- 172.21.15.150:0/1414290520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e0069800 msgr2=0x7f13e010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.659 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 --2- 172.21.15.150:0/1414290520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e0069800 0x7f13e010e670 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009cd0 tx=0x7f13d802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.659 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 -- 172.21.15.150:0/1414290520 shutdown_connections 2024-03-18T16:06:54.659 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 --2- 172.21.15.150:0/1414290520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e0069800 0x7f13e010e670 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 --2- 172.21.15.150:0/1414290520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0068e60 0x7f13e00692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 --2- 172.21.15.150:0/1414290520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e01078b0 0x7f13e0107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 -- 172.21.15.150:0/1414290520 >> 172.21.15.150:0/1414290520 conn(0x7f13e0077530 msgr2=0x7f13e0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 -- 172.21.15.150:0/1414290520 shutdown_connections 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.655+0000 7f13e5989640 1 -- 172.21.15.150:0/1414290520 wait complete. 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 Processor -- start 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 -- start start 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0068e60 0x7f13e0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0069800 0x7f13e0198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0111ba0 con 0x7f13e01078b0 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0111a20 con 0x7f13e0068e60 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13e5989640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0111d20 con 0x7f13e0069800 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55350/0 (socket says 172.21.15.150:55350) 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0069800 0x7f13e0198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13df7fe640 1 -- 172.21.15.150:0/371698897 learned_addr learned my addr 172.21.15.150:0/371698897 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.656+0000 7f13deffd640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0068e60 0x7f13e0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 -- 172.21.15.150:0/371698897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0069800 msgr2=0x7f13e0198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0069800 0x7f13e0198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 -- 172.21.15.150:0/371698897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0068e60 msgr2=0x7f13e0197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0068e60 0x7f13e0197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 -- 172.21.15.150:0/371698897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e019cf30 con 0x7f13e01078b0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13df7fe640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f13d8004af0 tx=0x7f13d8032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13e4987640 1 -- 172.21.15.150:0/371698897 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8042020 con 0x7f13e01078b0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13e4987640 1 -- 172.21.15.150:0/371698897 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d8032e80 con 0x7f13e01078b0 2024-03-18T16:06:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e019d1c0 con 0x7f13e01078b0 2024-03-18T16:06:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13e4987640 1 -- 172.21.15.150:0/371698897 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8039040 con 0x7f13e01078b0 2024-03-18T16:06:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.657+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e01a4ab0 con 0x7f13e01078b0 2024-03-18T16:06:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e4987640 1 -- 172.21.15.150:0/371698897 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f13d8045020 con 0x7f13e01078b0 2024-03-18T16:06:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 Processor -- start 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 -- start start 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0130 0x5574c0be0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0bebc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5574c0b44a40 con 0x5574c0be9820 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5574c0aed920 con 0x5574c0be0130 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e8579740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5574c0b40ae0 con 0x5574c0be0a50 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13e4987640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 0x7f13c80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13deffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0bebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13deffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:56696/0 (socket says 172.21.15.150:56696) 2024-03-18T16:06:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13deffd640 1 -- 172.21.15.150:0/1146425510 learned_addr learned my addr 172.21.15.150:0/1146425510 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.660+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0130 0x5574c0be0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13df7fe640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 0x7f13c80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13e4987640 1 -- 172.21.15.150:0/371698897 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f13d80f0b50 con 0x7f13e01078b0 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 -- 172.21.15.150:0/1146425510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0130 msgr2=0x5574c0be0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0130 0x5574c0be0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 -- 172.21.15.150:0/1146425510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 msgr2=0x5574c0bebc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0bebc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 -- 172.21.15.150:0/1146425510 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c0b05460 con 0x5574c0be0a50 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0bebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13deffd640 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009cf0 tx=0x7f13d0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7df052a6e1c522b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13df7fe640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 0x7f13c80a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009ca0 tx=0x7f13d802c0f0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13c5ffb640 1 -- 172.21.15.150:0/1146425510 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0004230 con 0x5574c0be0a50 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13c5ffb640 1 -- 172.21.15.150:0/1146425510 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d00043d0 con 0x5574c0be0a50 2024-03-18T16:06:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.661+0000 7f13c5ffb640 1 -- 172.21.15.150:0/1146425510 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0019840 con 0x5574c0be0a50 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- 172.21.15.150:0/1146425510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 msgr2=0x5574c0be92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009cf0 tx=0x7f13d0011810 comp rx=0 tx=0).stop 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- 172.21.15.150:0/1146425510 shutdown_connections 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0bebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0a50 0x5574c0be92e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 --2- 172.21.15.150:0/1146425510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0130 0x5574c0be0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- 172.21.15.150:0/1146425510 >> 172.21.15.150:0/1146425510 conn(0x5574c0bdaa50 msgr2=0x5574c0bdae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- 172.21.15.150:0/1146425510 shutdown_connections 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- 172.21.15.150:0/1146425510 wait complete. 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.662+0000 7f13e8579740 1 -- start start 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 Processor -- start 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5574c0bec450 con 0x5574c0be9820 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5574c0b00d70 con 0x5574c0be0a50 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5574c0bec5d0 con 0x5574c0be0130 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13e8579740 10 client.0 _ll_register_callbacks cb 0x5574c0858a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13de7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55374/0 (socket says 172.21.15.150:55374) 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:35278/0 (socket says 172.21.15.150:35278) 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13de7fc640 1 -- 172.21.15.150:0/2009106445 learned_addr learned my addr 172.21.15.150:0/2009106445 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13de7fc640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 -- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 msgr2=0x5574c0be6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 -- 172.21.15.150:0/2009106445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 msgr2=0x5574c0be71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13deffd640 1 -- 172.21.15.150:0/2009106445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c0859210 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.663+0000 7f13de7fc640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13deffd640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f13d001b040 tx=0x7f13d0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fb13c025af30f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0027070 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d00043d0 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f13d0005240 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f13d0033050 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 10 client.25742 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574c0925620 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d00043d0 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f13d0004ba0 con 0x5574c0be0a50 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 10 client.25742 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 20 client.25742 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0bfbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 4 client.25742 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.664+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c0b51f80 con 0x5574c0bf9850 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.665+0000 7f13de7fc640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0bfbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.665+0000 7f13beffd640 10 client.25742 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.665+0000 7f13de7fc640 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0bfbc40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=106a8fdb466b3a71 server_cookie=bc164009c1366f03 in_seq=0 out_seq=0 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13beffd640 1 -- 172.21.15.150:0/2009106445 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574c0b51f80 con 0x5574c0bf9850 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13beffd640 10 client.25742 handle_command_reply: tid=0 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 client.25742 shutdown 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 msgr2=0x5574c0be6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f13d001b040 tx=0x7f13d0003820 comp rx=0 tx=0).stop 2024-03-18T16:06:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 shutdown_connections 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0bfbc40 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13beffd640 0 client.25742 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be6cb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 --2- 172.21.15.150:0/2009106445 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 >> 172.21.15.150:0/2009106445 conn(0x5574c0bdaa50 msgr2=0x5574c0bdc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 shutdown_connections 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 1 -- 172.21.15.150:0/2009106445 wait complete. 2024-03-18T16:06:54.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.666+0000 7f13e8579740 20 client.25742 trim_cache size 0 max 16384 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 Processor -- start 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 -- start start 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be9820 0x5574c0be58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01afc50 con 0x5574c0be0a50 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01ad520 con 0x5574c0be9820 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13e8579740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01ab370 con 0x5574c0be0130 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13deffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13deffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55388/0 (socket says 172.21.15.150:55388) 2024-03-18T16:06:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13deffd640 1 -- 172.21.15.150:0/1069321731 learned_addr learned my addr 172.21.15.150:0/1069321731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.683+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be9820 0x5574c0be58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 -- 172.21.15.150:0/1069321731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 msgr2=0x5574c0be8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 -- 172.21.15.150:0/1069321731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be9820 msgr2=0x5574c0be58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be9820 0x5574c0be58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 -- 172.21.15.150:0/1069321731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c0bda880 con 0x5574c0be0a50 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13deffd640 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f13d0005970 tx=0x7f13d00059a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebc97d14436335e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13c57fa640 1 -- 172.21.15.150:0/1069321731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0027070 con 0x5574c0be0a50 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13c57fa640 1 -- 172.21.15.150:0/1069321731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d0003d80 con 0x5574c0be0a50 2024-03-18T16:06:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.684+0000 7f13c57fa640 1 -- 172.21.15.150:0/1069321731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0005260 con 0x5574c0be0a50 2024-03-18T16:06:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- 172.21.15.150:0/1069321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 msgr2=0x5574c0bf51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f13d0005970 tx=0x7f13d00059a0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- 172.21.15.150:0/1069321731 shutdown_connections 2024-03-18T16:06:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be9820 0x5574c0be58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be0a50 0x5574c0bf51f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 --2- 172.21.15.150:0/1069321731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- 172.21.15.150:0/1069321731 >> 172.21.15.150:0/1069321731 conn(0x5574c0bdaa50 msgr2=0x5574c0bd9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- 172.21.15.150:0/1069321731 shutdown_connections 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- 172.21.15.150:0/1069321731 wait complete. 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 -- start start 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.685+0000 7f13e8579740 1 Processor -- start 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01ab9e0 con 0x5574c0be9820 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01ace50 con 0x5574c0be0a50 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f13e01ac0b0 con 0x5574c0be0130 2024-03-18T16:06:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13e8579740 10 client.0 _ll_register_callbacks cb 0x7f13d4009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:35302/0 (socket says 172.21.15.150:35302) 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 -- 172.21.15.150:0/1130333243 learned_addr learned my addr 172.21.15.150:0/1130333243 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 -- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 msgr2=0x5574c0be3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 -- 172.21.15.150:0/1130333243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 msgr2=0x5574c0be3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 -- 172.21.15.150:0/1130333243 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c0c5ea70 con 0x5574c0be0a50 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13deffd640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f13d0011d10 tx=0x7f13d00099d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88dd790564f52406 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.686+0000 7f13beffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0027040 con 0x5574c0be0a50 2024-03-18T16:06:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d0003d80 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8483+0+0 (secure 0 0 0) 0x7f13d0005000 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f13d0009b70 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 10 client.25748 fetch_fsmap learned FSMap version 31 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5574c0c5fce0 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d0003d80 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f13d0023940 con 0x5574c0be0a50 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 10 client.25748 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 20 client.25748 populate_metadata read hostname 'smithi150' 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0c622d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 4 client.25748 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574c0b4c2d0 con 0x5574c0bf9850 2024-03-18T16:06:54.694 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.687+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0c622d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:54.695 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.688+0000 7f13beffd640 10 client.25748 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:54.695 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.688+0000 7f13de7fc640 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0c622d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=344973d5b382b6cb server_cookie=ba0649b07d896a86 in_seq=0 out_seq=0 2024-03-18T16:06:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13beffd640 1 -- 172.21.15.150:0/1130333243 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574c0b4c2d0 con 0x5574c0bf9850 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13beffd640 10 client.25748 handle_command_reply: tid=0 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 client.25748 shutdown 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 msgr2=0x5574c0be3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f13d0011d10 tx=0x7f13d00099d0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 shutdown_connections 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5574c0bf9850 0x5574c0c622d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5574c0be9820 0x5574c0be3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5574c0be0a50 0x5574c0be3770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 --2- 172.21.15.150:0/1130333243 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5574c0be0130 0x5574c0be3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13beffd640 0 client.25748 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 >> 172.21.15.150:0/1130333243 conn(0x5574c0bdaa50 msgr2=0x5574c0be6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.698+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 shutdown_connections 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e8579740 1 -- 172.21.15.150:0/1130333243 wait complete. 2024-03-18T16:06:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e8579740 20 client.25748 trim_cache size 0 max 16384 2024-03-18T16:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 msgr2=0x7f13c80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 0x7f13c80a4ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009ca0 tx=0x7f13d802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 msgr2=0x7f13e019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f13d8004af0 tx=0x7f13d8032c90 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 shutdown_connections 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f13c80a2a10 0x7f13c80a4ed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13e01078b0 0x7f13e019c800 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13e0069800 0x7f13e0198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 --2- 172.21.15.150:0/371698897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13e0068e60 0x7f13e0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 >> 172.21.15.150:0/371698897 conn(0x7f13e0077530 msgr2=0x7f13e00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 shutdown_connections 2024-03-18T16:06:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:06:54.699+0000 7f13e5989640 1 -- 172.21.15.150:0/371698897 wait complete. 2024-03-18T16:06:55.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 685997 11.02 MB/sec execute 220 sec latency 42.133 ms 2024-03-18T16:06:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:55 smithi191 ceph-mon[36784]: pgmap v210: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:55 smithi156 ceph-mon[35919]: pgmap v210: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:55 smithi150 ceph-mon[31946]: pgmap v210: 97 pgs: 97 active+clean; 431 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 185 op/s 2024-03-18T16:06:56.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 687848 11.01 MB/sec execute 221 sec latency 18.748 ms 2024-03-18T16:06:57.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 689659 11.01 MB/sec execute 222 sec latency 14.373 ms 2024-03-18T16:06:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:57 smithi191 ceph-mon[36784]: pgmap v211: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-03-18T16:06:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:57 smithi156 ceph-mon[35919]: pgmap v211: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-03-18T16:06:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:57 smithi150 ceph-mon[31946]: pgmap v211: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 280 op/s 2024-03-18T16:06:58.329 INFO:tasks.workunit.client.0.smithi150.stdout: 1 691544 11.00 MB/sec execute 223 sec latency 22.958 ms 2024-03-18T16:06:59.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 693282 11.00 MB/sec execute 224 sec latency 28.893 ms 2024-03-18T16:06:59.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:06:59 smithi191 ceph-mon[36784]: pgmap v212: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-18T16:07:00.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:06:59 smithi156 ceph-mon[35919]: pgmap v212: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-18T16:07:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:06:59 smithi150 ceph-mon[31946]: pgmap v212: 97 pgs: 97 active+clean; 438 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-18T16:07:00.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 695087 10.98 MB/sec execute 225 sec latency 60.068 ms 2024-03-18T16:07:01.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 696472 10.99 MB/sec execute 226 sec latency 25.633 ms 2024-03-18T16:07:01.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:01 smithi191 ceph-mon[36784]: pgmap v213: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-03-18T16:07:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:01 smithi156 ceph-mon[35919]: pgmap v213: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-03-18T16:07:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:01 smithi150 ceph-mon[31946]: pgmap v213: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 288 op/s 2024-03-18T16:07:02.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 697735 10.96 MB/sec execute 227 sec latency 115.949 ms 2024-03-18T16:07:03.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 699860 10.97 MB/sec execute 228 sec latency 28.689 ms 2024-03-18T16:07:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:03 smithi191 ceph-mon[36784]: pgmap v214: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 198 op/s 2024-03-18T16:07:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:03 smithi191 ceph-mon[36784]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:07:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:03 smithi156 ceph-mon[35919]: pgmap v214: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 198 op/s 2024-03-18T16:07:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:03 smithi156 ceph-mon[35919]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:07:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:03 smithi150 ceph-mon[31946]: pgmap v214: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 198 op/s 2024-03-18T16:07:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:03 smithi150 ceph-mon[31946]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:07:04.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 701242 10.95 MB/sec execute 229 sec latency 25.085 ms 2024-03-18T16:07:05.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 703216 10.95 MB/sec execute 230 sec latency 104.046 ms 2024-03-18T16:07:05.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:05 smithi191 ceph-mon[36784]: pgmap v216: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 237 op/s 2024-03-18T16:07:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:05 smithi156 ceph-mon[35919]: pgmap v216: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 237 op/s 2024-03-18T16:07:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:05 smithi150 ceph-mon[31946]: pgmap v216: 97 pgs: 97 active+clean; 416 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 237 op/s 2024-03-18T16:07:06.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 704623 10.94 MB/sec execute 231 sec latency 34.273 ms 2024-03-18T16:07:07.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 706753 10.94 MB/sec execute 232 sec latency 24.046 ms 2024-03-18T16:07:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 -- 172.21.15.150:0/1463871674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381067e0 msgr2=0x7f4638106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 --2- 172.21.15.150:0/1463871674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381067e0 0x7f4638106bc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4628009900 tx=0x7f462802bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 -- 172.21.15.150:0/1463871674 shutdown_connections 2024-03-18T16:07:07.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 --2- 172.21.15.150:0/1463871674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f463810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 --2- 172.21.15.150:0/1463871674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381009c0 0x7f4638100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 --2- 172.21.15.150:0/1463871674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381067e0 0x7f4638106bc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 -- 172.21.15.150:0/1463871674 >> 172.21.15.150:0/1463871674 conn(0x7f46380774b0 msgr2=0x7f46380778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 -- 172.21.15.150:0/1463871674 shutdown_connections 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.793+0000 7f463e789640 1 -- 172.21.15.150:0/1463871674 wait complete. 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 Processor -- start 2024-03-18T16:07:07.796 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 -- start start 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f46381984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 0x7f463819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4638111bc0 con 0x7f46381009c0 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4638111a40 con 0x7f46381067e0 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463e789640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4638111d40 con 0x7f4638101380 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f4637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f4637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37978/0 (socket says 172.21.15.150:37978) 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f4637fff640 1 -- 172.21.15.150:0/1231139716 learned_addr learned my addr 172.21.15.150:0/1231139716 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f46377fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f46381984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.794+0000 7f463cd26640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 0x7f463819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 -- 172.21.15.150:0/1231139716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 msgr2=0x7f46381984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f46381984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 -- 172.21.15.150:0/1231139716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 msgr2=0x7f463819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 0x7f463819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 -- 172.21.15.150:0/1231139716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463819cf70 con 0x7f46381009c0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f463cd26640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 0x7f463819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f46377fe640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f46381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f4637fff640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f462802efd0 tx=0x7f46280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f46357fa640 1 -- 172.21.15.150:0/1231139716 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628041070 con 0x7f46381009c0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f46357fa640 1 -- 172.21.15.150:0/1231139716 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f462802ccc0 con 0x7f46381009c0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463819d200 con 0x7f46381009c0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f46357fa640 1 -- 172.21.15.150:0/1231139716 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628003d10 con 0x7f46381009c0 2024-03-18T16:07:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.795+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46381a4af0 con 0x7f46381009c0 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.797+0000 7f46357fa640 1 -- 172.21.15.150:0/1231139716 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f4628044070 con 0x7f46381009c0 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 Processor -- start 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 -- start start 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 0x559907506510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55990746aa40 con 0x55990750f820 2024-03-18T16:07:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559907413920 con 0x559907506a50 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f464137b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559907466ae0 con 0x559907506130 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f46357fa640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 0x7f46180a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f4637fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f46377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f4637fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:44772/0 (socket says 172.21.15.150:44772) 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f46377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37992/0 (socket says 172.21.15.150:37992) 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f4637fff640 1 -- 172.21.15.150:0/1717100120 learned_addr learned my addr 172.21.15.150:0/1717100120 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f463cd26640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 0x559907506510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.798+0000 7f463cd26640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 0x7f46180a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f46357fa640 1 -- 172.21.15.150:0/1231139716 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f46280f0700 con 0x7f46381009c0 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 -- 172.21.15.150:0/1717100120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 msgr2=0x559907506510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 0x559907506510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 -- 172.21.15.150:0/1717100120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 msgr2=0x559907511c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 -- 172.21.15.150:0/1717100120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55990742b460 con 0x559907506a50 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f463cd26640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 0x559907506510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f46377fe640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f4637fff640 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4628000c00 tx=0x7f462802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397e5fcbc98dd33e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f463cd26640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 0x7f46180a4f50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46300058c0 tx=0x7f4630005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f462e7fc640 1 -- 172.21.15.150:0/1717100120 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462804b070 con 0x559907506a50 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f462e7fc640 1 -- 172.21.15.150:0/1717100120 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46280c0f90 con 0x559907506a50 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.799+0000 7f462e7fc640 1 -- 172.21.15.150:0/1717100120 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280b9560 con 0x559907506a50 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- 172.21.15.150:0/1717100120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 msgr2=0x55990750f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4628000c00 tx=0x7f462802bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- 172.21.15.150:0/1717100120 shutdown_connections 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55990750f820 0x559907511c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750f2e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 --2- 172.21.15.150:0/1717100120 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559907506130 0x559907506510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- 172.21.15.150:0/1717100120 >> 172.21.15.150:0/1717100120 conn(0x559907500a50 msgr2=0x559907500e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- 172.21.15.150:0/1717100120 shutdown_connections 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- 172.21.15.150:0/1717100120 wait complete. 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.800+0000 7f464137b740 1 -- start start 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 Processor -- start 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x55990750c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990750d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559907413920 con 0x559907506130 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559907426d70 con 0x559907506a50 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559907512210 con 0x55990750f820 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f464137b740 10 client.0 _ll_register_callbacks cb 0x55990717ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x55990750c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f463cd26640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990750d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:44774/0 (socket says 172.21.15.150:44774) 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46377fe640 1 -- 172.21.15.150:0/2414830254 learned_addr learned my addr 172.21.15.150:0/2414830254 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 -- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 msgr2=0x55990750d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990750d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 -- 172.21.15.150:0/2414830254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 msgr2=0x55990750ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 -- 172.21.15.150:0/2414830254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55990717f210 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46377fe640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f4637fff640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x55990750c910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f46280edf90 tx=0x7f4628003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb4234698235ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46277fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.801+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280bf030 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f46280035f0 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46280c0f90 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f462804b050 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 10 client.15996 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55990724b620 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46280c0f90 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f46280bd150 con 0x559907506130 2024-03-18T16:07:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 10 client.15996 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 20 client.15996 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990751f530 0x559907521920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 4 client.15996 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559907477f80 con 0x55990751f530 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.802+0000 7f46377fe640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990751f530 0x559907521920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f46377fe640 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990751f530 0x559907521920 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ff4d5a40657a21 server_cookie=1570e77979166d44 in_seq=0 out_seq=0 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f46277fe640 10 client.15996 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f46277fe640 1 -- 172.21.15.150:0/2414830254 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559907477f80 con 0x55990751f530 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f46277fe640 10 client.15996 handle_command_reply: tid=0 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 client.15996 shutdown 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 msgr2=0x55990750c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x55990750c910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f46280edf90 tx=0x7f4628003840 comp rx=0 tx=0).stop 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 shutdown_connections 2024-03-18T16:07:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990751f530 0x559907521920 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990750d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f46277fe640 0 client.15996 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 --2- 172.21.15.150:0/2414830254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x55990750c910 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.803+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 >> 172.21.15.150:0/2414830254 conn(0x559907500a50 msgr2=0x5599075027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.804+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 shutdown_connections 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.804+0000 7f464137b740 1 -- 172.21.15.150:0/2414830254 wait complete. 2024-03-18T16:07:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.804+0000 7f464137b740 20 client.15996 trim_cache size 0 max 16384 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 Processor -- start 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 -- start start 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506130 0x55990750f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990751ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46381afcb0 con 0x559907506a50 2024-03-18T16:07:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46381ad580 con 0x559907506130 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f464137b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f46381ab3f0 con 0x55990750f820 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38016/0 (socket says 172.21.15.150:38016) 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 -- 172.21.15.150:0/3406664965 learned_addr learned my addr 172.21.15.150:0/3406664965 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f463cd26640 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990751ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f4637fff640 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506130 0x55990750f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 -- 172.21.15.150:0/3406664965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 msgr2=0x55990751ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990751ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 -- 172.21.15.150:0/3406664965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506130 msgr2=0x55990750f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506130 0x55990750f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 -- 172.21.15.150:0/3406664965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599075055d0 con 0x559907506a50 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f46377fe640 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4620004a80 tx=0x7f4620009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926e572b4a82d6d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f462dffb640 1 -- 172.21.15.150:0/3406664965 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620025070 con 0x559907506a50 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f462dffb640 1 -- 172.21.15.150:0/3406664965 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46200205d0 con 0x559907506a50 2024-03-18T16:07:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.819+0000 7f462dffb640 1 -- 172.21.15.150:0/3406664965 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620004500 con 0x559907506a50 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- 172.21.15.150:0/3406664965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 msgr2=0x559907517840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4620004a80 tx=0x7f4620009500 comp rx=0 tx=0).stop 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- 172.21.15.150:0/3406664965 shutdown_connections 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x55990751ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506a50 0x559907517840 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 --2- 172.21.15.150:0/3406664965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506130 0x55990750f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- 172.21.15.150:0/3406664965 >> 172.21.15.150:0/3406664965 conn(0x559907500a50 msgr2=0x559907501e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- 172.21.15.150:0/3406664965 shutdown_connections 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- 172.21.15.150:0/3406664965 wait complete. 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.820+0000 7f464137b740 1 -- start start 2024-03-18T16:07:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 Processor -- start 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x559907508b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46381aba40 con 0x559907506130 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f46381aceb0 con 0x559907506a50 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f46381ac110 con 0x55990750f820 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f464137b740 10 client.0 _ll_register_callbacks cb 0x7f4620009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38022/0 (socket says 172.21.15.150:38022) 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 -- 172.21.15.150:0/2625386016 learned_addr learned my addr 172.21.15.150:0/2625386016 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f463cd26640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x559907508b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f46377fe640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 -- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 msgr2=0x559907508b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x559907508b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 -- 172.21.15.150:0/2625386016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 msgr2=0x55990750a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 -- 172.21.15.150:0/2625386016 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599075093b0 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f4637fff640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f462802f020 tx=0x7f46280f0a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570d7ab90fb3845e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f46277fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.821+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280c6e60 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8571+0+0 (secure 0 0 0) 0x7f46280bd150 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46200205d0 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f46280bf070 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 10 client.16002 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599075093b0 con 0x559907506130 2024-03-18T16:07:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46200205d0 con 0x559907506130 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f46280b9560 con 0x559907506130 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 10 client.16002 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 20 client.16002 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990758a620 0x55990758ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 4 client.16002 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5599074722d0 con 0x55990758a620 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.822+0000 7f46377fe640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990758a620 0x55990758ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.823+0000 7f46277fe640 10 client.16002 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.823+0000 7f46377fe640 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990758a620 0x55990758ca10 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=872c7cd2d6d3ae server_cookie=d0a3a05e0a5fdbac in_seq=0 out_seq=0 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f46277fe640 1 -- 172.21.15.150:0/2625386016 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5599074722d0 con 0x55990758a620 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f46277fe640 10 client.16002 handle_command_reply: tid=0 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 client.16002 shutdown 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 msgr2=0x559907509f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f462802f020 tx=0x7f46280f0a00 comp rx=0 tx=0).stop 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 shutdown_connections 2024-03-18T16:07:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55990758a620 0x55990758ca10 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55990750f820 0x559907508b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559907506a50 0x55990750a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f46277fe640 0 client.16002 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 --2- 172.21.15.150:0/2625386016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559907506130 0x559907509f40 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 >> 172.21.15.150:0/2625386016 conn(0x559907500a50 msgr2=0x5599074ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 shutdown_connections 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 1 -- 172.21.15.150:0/2625386016 wait complete. 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.834+0000 7f464137b740 20 client.16002 trim_cache size 0 max 16384 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:07:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 msgr2=0x7f46180a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 0x7f46180a4f50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46300058c0 tx=0x7f4630005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 msgr2=0x7f4638197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f462802efd0 tx=0x7f46280042f0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 shutdown_connections 2024-03-18T16:07:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f46180a2a90 0x7f46180a4f50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46381067e0 0x7f463819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4638101380 0x7f46381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 --2- 172.21.15.150:0/1231139716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46381009c0 0x7f4638197f80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 >> 172.21.15.150:0/1231139716 conn(0x7f46380774b0 msgr2=0x7f46380ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 shutdown_connections 2024-03-18T16:07:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:07.835+0000 7f463e789640 1 -- 172.21.15.150:0/1231139716 wait complete. 2024-03-18T16:07:07.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:07 smithi191 ceph-mon[36784]: pgmap v217: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:07 smithi156 ceph-mon[35919]: pgmap v217: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:07 smithi150 ceph-mon[31946]: pgmap v217: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:08.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 708205 10.92 MB/sec execute 233 sec latency 16.214 ms 2024-03-18T16:07:09.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 710490 10.94 MB/sec execute 234 sec latency 17.754 ms 2024-03-18T16:07:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:09 smithi191 ceph-mon[36784]: pgmap v218: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:09 smithi191 ceph-mon[36784]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:07:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:09 smithi156 ceph-mon[35919]: pgmap v218: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:09 smithi156 ceph-mon[35919]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:07:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:09 smithi150 ceph-mon[31946]: pgmap v218: 97 pgs: 97 active+clean; 423 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 249 op/s 2024-03-18T16:07:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:09 smithi150 ceph-mon[31946]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:07:10.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 711753 10.91 MB/sec execute 235 sec latency 76.079 ms 2024-03-18T16:07:11.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 713873 10.92 MB/sec execute 236 sec latency 55.748 ms 2024-03-18T16:07:11.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:11 smithi191 ceph-mon[36784]: pgmap v220: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 279 op/s 2024-03-18T16:07:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:11 smithi156 ceph-mon[35919]: pgmap v220: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 279 op/s 2024-03-18T16:07:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:11 smithi150 ceph-mon[31946]: pgmap v220: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 279 op/s 2024-03-18T16:07:12.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 715324 10.90 MB/sec execute 237 sec latency 41.597 ms 2024-03-18T16:07:13.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 717599 10.91 MB/sec execute 238 sec latency 17.156 ms 2024-03-18T16:07:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:13 smithi191 ceph-mon[36784]: pgmap v221: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 226 op/s 2024-03-18T16:07:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:13 smithi156 ceph-mon[35919]: pgmap v221: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 226 op/s 2024-03-18T16:07:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:13 smithi150 ceph-mon[31946]: pgmap v221: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 226 op/s 2024-03-18T16:07:14.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 719089 10.89 MB/sec execute 239 sec latency 24.657 ms 2024-03-18T16:07:15.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 721134 10.90 MB/sec execute 240 sec latency 92.213 ms 2024-03-18T16:07:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:15 smithi191 ceph-mon[36784]: pgmap v222: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 223 op/s 2024-03-18T16:07:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:15 smithi156 ceph-mon[35919]: pgmap v222: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 223 op/s 2024-03-18T16:07:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:15 smithi150 ceph-mon[31946]: pgmap v222: 97 pgs: 97 active+clean; 436 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 223 op/s 2024-03-18T16:07:16.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 722632 10.88 MB/sec execute 241 sec latency 18.071 ms 2024-03-18T16:07:16.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:16 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:17.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:16 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:17.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:16 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:17.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 724715 10.89 MB/sec execute 242 sec latency 31.608 ms 2024-03-18T16:07:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:17 smithi191 ceph-mon[36784]: pgmap v223: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:17 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:07:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:17 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:07:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:17 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:07:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:17 smithi156 ceph-mon[35919]: pgmap v223: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:17 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:07:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:17 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:07:18.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:17 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:07:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:17 smithi150 ceph-mon[31946]: pgmap v223: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:17 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:07:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:17 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:07:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:17 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:07:18.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 725998 10.87 MB/sec execute 243 sec latency 169.412 ms 2024-03-18T16:07:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:18 smithi156 ceph-mon[35919]: pgmap v224: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:18 smithi150 ceph-mon[31946]: pgmap v224: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:19.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:18 smithi191 ceph-mon[36784]: pgmap v224: 97 pgs: 97 active+clean; 440 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 221 op/s 2024-03-18T16:07:19.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 728222 10.88 MB/sec execute 244 sec latency 21.604 ms 2024-03-18T16:07:20.330 INFO:tasks.workunit.client.0.smithi150.stdout: 1 729605 10.86 MB/sec execute 245 sec latency 67.801 ms 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 -- 172.21.15.150:0/1422505701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc105680 msgr2=0x7fd6cc105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1422505701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc105680 0x7fd6cc105a60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009860 tx=0x7fd6bc02bb70 comp rx=0 tx=0).stop 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 -- 172.21.15.150:0/1422505701 shutdown_connections 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1422505701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 0x7fd6cc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1422505701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc100930 0x7fd6cc100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1422505701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc105680 0x7fd6cc105a60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 -- 172.21.15.150:0/1422505701 >> 172.21.15.150:0/1422505701 conn(0x7fd6cc077530 msgr2=0x7fd6cc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 -- 172.21.15.150:0/1422505701 shutdown_connections 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.926+0000 7fd6d395b640 1 -- 172.21.15.150:0/1422505701 wait complete. 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 Processor -- start 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 -- start start 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 0x7fd6cc1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc105680 0x7fd6cc1a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc110b60 con 0x7fd6cc100930 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1109e0 con 0x7fd6cc1012f0 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d395b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc110ce0 con 0x7fd6cc105680 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60482/0 (socket says 172.21.15.150:60482) 2024-03-18T16:07:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 -- 172.21.15.150:0/1515065641 learned_addr learned my addr 172.21.15.150:0/1515065641 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc105680 0x7fd6cc1a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 0x7fd6cc1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 -- 172.21.15.150:0/1515065641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc105680 msgr2=0x7fd6cc1a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc105680 0x7fd6cc1a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 -- 172.21.15.150:0/1515065641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 msgr2=0x7fd6cc1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 0x7fd6cc1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 -- 172.21.15.150:0/1515065641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc1a4460 con 0x7fd6cc100930 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.927+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc037040 tx=0x7fd6bc0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.928+0000 7fd6c27fc640 1 -- 172.21.15.150:0/1515065641 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc041020 con 0x7fd6cc100930 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.928+0000 7fd6c27fc640 1 -- 172.21.15.150:0/1515065641 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc02c9f0 con 0x7fd6cc100930 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.928+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc1a4630 con 0x7fd6cc100930 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.928+0000 7fd6c27fc640 1 -- 172.21.15.150:0/1515065641 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc039040 con 0x7fd6cc100930 2024-03-18T16:07:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.928+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1a4aa0 con 0x7fd6cc100930 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.930+0000 7fd6c27fc640 1 -- 172.21.15.150:0/1515065641 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fd6bc044020 con 0x7fd6cc100930 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.930+0000 7fd6d6624740 1 Processor -- start 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.930+0000 7fd6d6624740 1 -- start start 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x562203476510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203481c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5622033daa40 con 0x562203476a50 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562203383920 con 0x56220347f820 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d6624740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5622033d6ae0 con 0x562203476130 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d1ef8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x562203476510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d1ef8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x562203476510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:52868/0 (socket says 172.21.15.150:52868) 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60494/0 (socket says 172.21.15.150:60494) 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d1ef8640 1 -- 172.21.15.150:0/2433813764 learned_addr learned my addr 172.21.15.150:0/2433813764 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203481c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6c27fc640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 0x7fd6b80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 0x7fd6b80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6c27fc640 1 -- 172.21.15.150:0/1515065641 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fd6bc0f05a0 con 0x7fd6cc100930 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 -- 172.21.15.150:0/2433813764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 msgr2=0x562203476510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x562203476510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 -- 172.21.15.150:0/2433813764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 msgr2=0x56220347f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 -- 172.21.15.150:0/2433813764 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220339b460 con 0x56220347f820 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.931+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203481c10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005fd0 tx=0x7fd6c80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c16dba6312f21d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 0x7fd6b80a4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009fd0 tx=0x7fd6c4004ab0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6b77fe640 1 -- 172.21.15.150:0/2433813764 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8012450 con 0x56220347f820 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6b77fe640 1 -- 172.21.15.150:0/2433813764 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c8012ad0 con 0x56220347f820 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6b77fe640 1 -- 172.21.15.150:0/2433813764 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800fbc0 con 0x56220347f820 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 -- 172.21.15.150:0/2433813764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 msgr2=0x562203481c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203481c10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005fd0 tx=0x7fd6c80117b0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 -- 172.21.15.150:0/2433813764 shutdown_connections 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203481c10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476a50 0x56220347f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2433813764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x562203476510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.932+0000 7fd6d6624740 1 -- 172.21.15.150:0/2433813764 >> 172.21.15.150:0/2433813764 conn(0x562203470a50 msgr2=0x562203470e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- 172.21.15.150:0/2433813764 shutdown_connections 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- 172.21.15.150:0/2433813764 wait complete. 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- start start 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 Processor -- start 2024-03-18T16:07:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x56220347ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x56220347d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562203482450 con 0x562203476130 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562203396d70 con 0x56220347f820 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5622034825d0 con 0x562203476a50 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d6624740 10 client.0 _ll_register_callbacks cb 0x5622030eea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60510/0 (socket says 172.21.15.150:60510) 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2240062945 learned_addr learned my addr 172.21.15.150:0/2240062945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x56220347ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.933+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x56220347d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 msgr2=0x56220347ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x56220347ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2240062945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 msgr2=0x56220347d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x56220347d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2240062945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622030ef210 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b96e0 tx=0x7fd6bc009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d46d4cd95873e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0348d0 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fd6bc032db0 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c8012ad0 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6bc0bd410 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d6624740 10 client.16014 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5622031bb620 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c8012ad0 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd6bc0d5020 con 0x562203476130 2024-03-18T16:07:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.934+0000 7fd6d6624740 10 client.16014 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d6624740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d6624740 20 client.16014 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x56220348f8f0 0x562203491d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d6624740 4 client.16014 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622033e7f80 con 0x56220348f8f0 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x56220348f8f0 0x562203491d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6b4ff9640 10 client.16014 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.935+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x56220348f8f0 0x562203491d40 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=201f8e5deef9a3c6 server_cookie=aee093ddf89818be in_seq=0 out_seq=0 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2240062945 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622033e7f80 con 0x56220348f8f0 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6b4ff9640 10 client.16014 handle_command_reply: tid=0 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 client.16014 shutdown 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 msgr2=0x56220347c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b96e0 tx=0x7fd6bc009490 comp rx=0 tx=0).stop 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 shutdown_connections 2024-03-18T16:07:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x56220348f8f0 0x562203491d40 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x56220347d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6b4ff9640 0 client.16014 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x56220347ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2240062945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x56220347c7b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 >> 172.21.15.150:0/2240062945 conn(0x562203470a50 msgr2=0x5622034727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 shutdown_connections 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 1 -- 172.21.15.150:0/2240062945 wait complete. 2024-03-18T16:07:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.936+0000 7fd6d6624740 20 client.16014 trim_cache size 0 max 16384 2024-03-18T16:07:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.951+0000 7fd6d6624740 1 Processor -- start 2024-03-18T16:07:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.951+0000 7fd6d6624740 1 -- start start 2024-03-18T16:07:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x56220347eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.954 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562203476a50 0x56220348b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1afd30 con 0x56220347f820 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1ad600 con 0x562203476a50 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d6624740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1ab470 con 0x562203476130 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x56220347eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60514/0 (socket says 172.21.15.150:60514) 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 -- 172.21.15.150:0/1680769635 learned_addr learned my addr 172.21.15.150:0/1680769635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562203476a50 0x56220348b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 -- 172.21.15.150:0/1680769635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 msgr2=0x56220347eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x56220347eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 -- 172.21.15.150:0/1680769635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562203476a50 msgr2=0x56220348b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.955 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562203476a50 0x56220348b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 -- 172.21.15.150:0/1680769635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562203470880 con 0x56220347f820 2024-03-18T16:07:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4013430 tx=0x7fd6c4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6ce0821a5c6c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6b6ffd640 1 -- 172.21.15.150:0/1680769635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4023be0 con 0x56220347f820 2024-03-18T16:07:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6b6ffd640 1 -- 172.21.15.150:0/1680769635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c4023d80 con 0x56220347f820 2024-03-18T16:07:20.956 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.952+0000 7fd6b6ffd640 1 -- 172.21.15.150:0/1680769635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c402b640 con 0x56220347f820 2024-03-18T16:07:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 -- 172.21.15.150:0/1680769635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 msgr2=0x5622034975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4013430 tx=0x7fd6c4005bd0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 -- 172.21.15.150:0/1680769635 shutdown_connections 2024-03-18T16:07:20.959 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56220347f820 0x5622034975f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562203476a50 0x56220348b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 --2- 172.21.15.150:0/1680769635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476130 0x56220347eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 -- 172.21.15.150:0/1680769635 >> 172.21.15.150:0/1680769635 conn(0x562203470a50 msgr2=0x562203477aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 -- 172.21.15.150:0/1680769635 shutdown_connections 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.953+0000 7fd6d6624740 1 -- 172.21.15.150:0/1680769635 wait complete. 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 -- start start 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 Processor -- start 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x5622034835a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203483ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1abac0 con 0x562203476130 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1acf30 con 0x56220347f820 2024-03-18T16:07:20.960 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1ac190 con 0x562203476a50 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d6624740 10 client.0 _ll_register_callbacks cb 0x7fd6bc005a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60518/0 (socket says 172.21.15.150:60518) 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2841093876 learned_addr learned my addr 172.21.15.150:0/2841093876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x5622034835a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d1ef8640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203483ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 msgr2=0x5622034835a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x5622034835a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2841093876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 msgr2=0x562203483ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203483ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.954+0000 7fd6d16f7640 1 -- 172.21.15.150:0/2841093876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622034f4c20 con 0x562203476130 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d16f7640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b9170 tx=0x7fd6bc0f07a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6811b5c4252389a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0da060 con 0x562203476130 2024-03-18T16:07:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fd6bc0f00f0 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c4023d80 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6bc0b9510 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 10 client.16020 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5622034f5e70 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c4023d80 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd6bc0d5040 con 0x562203476130 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 10 client.16020 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 20 client.16020 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5622034f6430 0x5622034f8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 4 client.16020 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.955+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622033e22d0 con 0x5622034f6430 2024-03-18T16:07:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.956+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5622034f6430 0x5622034f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.956+0000 7fd6b4ff9640 10 client.16020 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:20.963 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.956+0000 7fd6d0ef6640 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5622034f6430 0x5622034f8850 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebeb57a09fe23a06 server_cookie=5a6a92b32be1a5bb in_seq=0 out_seq=0 2024-03-18T16:07:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.973+0000 7fd6b4ff9640 1 -- 172.21.15.150:0/2841093876 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622033e22d0 con 0x5622034f6430 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.973+0000 7fd6b4ff9640 10 client.16020 handle_command_reply: tid=0 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.973+0000 7fd6d6624740 1 client.16020 shutdown 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 msgr2=0x562203483060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b9170 tx=0x7fd6bc0f07a0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 shutdown_connections 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5622034f6430 0x5622034f8850 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56220347f820 0x562203483ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6b4ff9640 0 client.16020 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562203476a50 0x5622034835a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 --2- 172.21.15.150:0/2841093876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562203476130 0x562203483060 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 >> 172.21.15.150:0/2841093876 conn(0x562203470a50 msgr2=0x562203471ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 shutdown_connections 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 1 -- 172.21.15.150:0/2841093876 wait complete. 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d6624740 20 client.16020 trim_cache size 0 max 16384 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:07:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 msgr2=0x7fd6b80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 0x7fd6b80a4ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009fd0 tx=0x7fd6c4004ab0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 msgr2=0x7fd6cc198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc037040 tx=0x7fd6bc0045b0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 shutdown_connections 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd6b80a2a10 0x7fd6b80a4ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6cc105680 0x7fd6cc1a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc1012f0 0x7fd6cc1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 --2- 172.21.15.150:0/1515065641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc100930 0x7fd6cc198ea0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.974+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 >> 172.21.15.150:0/1515065641 conn(0x7fd6cc077530 msgr2=0x7fd6cc0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.975+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 shutdown_connections 2024-03-18T16:07:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:20.975+0000 7fd6d395b640 1 -- 172.21.15.150:0/1515065641 wait complete. 2024-03-18T16:07:21.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 731787 10.87 MB/sec execute 246 sec latency 39.325 ms 2024-03-18T16:07:21.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:21 smithi191 ceph-mon[36784]: pgmap v225: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-03-18T16:07:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:21 smithi156 ceph-mon[35919]: pgmap v225: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-03-18T16:07:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:21 smithi150 ceph-mon[31946]: pgmap v225: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-03-18T16:07:22.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 733272 10.85 MB/sec execute 247 sec latency 20.400 ms 2024-03-18T16:07:23.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 735460 10.86 MB/sec execute 248 sec latency 13.190 ms 2024-03-18T16:07:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:23 smithi191 ceph-mon[36784]: pgmap v226: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:23 smithi156 ceph-mon[35919]: pgmap v226: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:23 smithi150 ceph-mon[31946]: pgmap v226: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:24.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 737366 10.84 MB/sec execute 249 sec latency 18.685 ms 2024-03-18T16:07:25.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 739031 10.85 MB/sec execute 250 sec latency 81.277 ms 2024-03-18T16:07:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:25 smithi191 ceph-mon[36784]: pgmap v227: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:25 smithi156 ceph-mon[35919]: pgmap v227: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:25 smithi150 ceph-mon[31946]: pgmap v227: 97 pgs: 97 active+clean; 456 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 184 op/s 2024-03-18T16:07:26.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 740652 10.83 MB/sec execute 251 sec latency 42.877 ms 2024-03-18T16:07:27.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 742711 10.84 MB/sec execute 252 sec latency 13.141 ms 2024-03-18T16:07:27.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:27 smithi191 ceph-mon[36784]: pgmap v228: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 279 op/s 2024-03-18T16:07:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:27 smithi156 ceph-mon[35919]: pgmap v228: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 279 op/s 2024-03-18T16:07:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:27 smithi150 ceph-mon[31946]: pgmap v228: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 279 op/s 2024-03-18T16:07:28.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 744584 10.82 MB/sec execute 253 sec latency 19.525 ms 2024-03-18T16:07:29.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 746416 10.83 MB/sec execute 254 sec latency 19.977 ms 2024-03-18T16:07:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:29 smithi191 ceph-mon[36784]: pgmap v229: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 176 op/s 2024-03-18T16:07:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:29 smithi156 ceph-mon[35919]: pgmap v229: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 176 op/s 2024-03-18T16:07:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:29 smithi150 ceph-mon[31946]: pgmap v229: 97 pgs: 97 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 176 op/s 2024-03-18T16:07:30.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 748266 10.82 MB/sec execute 255 sec latency 80.410 ms 2024-03-18T16:07:31.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 750027 10.82 MB/sec execute 256 sec latency 29.975 ms 2024-03-18T16:07:31.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:31 smithi191 ceph-mon[36784]: pgmap v230: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-18T16:07:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:31 smithi156 ceph-mon[35919]: pgmap v230: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-18T16:07:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:31 smithi150 ceph-mon[31946]: pgmap v230: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 273 op/s 2024-03-18T16:07:32.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 752216 10.83 MB/sec execute 257 sec latency 17.173 ms 2024-03-18T16:07:33.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 753827 10.82 MB/sec execute 258 sec latency 19.409 ms 2024-03-18T16:07:33.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:33 smithi191 ceph-mon[36784]: pgmap v231: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:33 smithi156 ceph-mon[35919]: pgmap v231: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 -- 172.21.15.150:0/2995867929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc100930 msgr2=0x7fa7bc100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/2995867929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc100930 0x7fa7bc100db0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009900 tx=0x7fa7a402bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 -- 172.21.15.150:0/2995867929 shutdown_connections 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/2995867929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/2995867929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc100930 0x7fa7bc100db0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/2995867929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 0x7fa7bc105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 -- 172.21.15.150:0/2995867929 >> 172.21.15.150:0/2995867929 conn(0x7fa7bc077530 msgr2=0x7fa7bc077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 -- 172.21.15.150:0/2995867929 shutdown_connections 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.064+0000 7fa7c14c6640 1 -- 172.21.15.150:0/2995867929 wait complete. 2024-03-18T16:07:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 Processor -- start 2024-03-18T16:07:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 -- start start 2024-03-18T16:07:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 0x7fa7bc19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc110ba0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc110a20 con 0x7fa7bc100930 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7c14c6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc110d20 con 0x7fa7bc105680 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7baffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7baffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:56932/0 (socket says 172.21.15.150:56932) 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7baffd640 1 -- 172.21.15.150:0/980486650 learned_addr learned my addr 172.21.15.150:0/980486650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 0x7fa7bc19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/980486650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 msgr2=0x7fa7bc19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 0x7fa7bc19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/980486650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 msgr2=0x7fa7bc196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/980486650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc19bf50 con 0x7fa7bc1012f0 2024-03-18T16:07:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.065+0000 7fa7baffd640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc1974a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a40042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7b3fff640 1 -- 172.21.15.150:0/980486650 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4041070 con 0x7fa7bc1012f0 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7b3fff640 1 -- 172.21.15.150:0/980486650 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a402ccc0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc19c1e0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7b3fff640 1 -- 172.21.15.150:0/980486650 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4003db0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.066+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc1a3ad0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.068+0000 7fa7b3fff640 1 -- 172.21.15.150:0/980486650 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fa7a4044070 con 0x7fa7bc1012f0 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.068+0000 7fa7c4191740 1 Processor -- start 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.068+0000 7fa7c4191740 1 -- start start 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7b3fff640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 0x7fa7a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c5b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c5be20 0x55df37c5e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55df37bb2a40 con 0x55df37c4fc70 2024-03-18T16:07:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55df37b5b920 con 0x55df37c5be20 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7c4191740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55df37baeae0 con 0x55df37c50590 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7baffd640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 0x7fa7a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c5b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37374/0 (socket says 172.21.15.150:37374) 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1099756643 learned_addr learned my addr 172.21.15.150:0/1099756643 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7b3fff640 1 -- 172.21.15.150:0/980486650 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fa7a40f06d0 con 0x7fa7bc1012f0 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c5be20 0x55df37c5e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1099756643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c5be20 msgr2=0x55df37c5e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c5be20 0x55df37c5e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1099756643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 msgr2=0x55df37c50050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1099756643 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df37b73460 con 0x55df37c50590 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7baffd640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 0x7fa7a80a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40049d0 tx=0x7fa7b4009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.069+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c5b8e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40edf30 tx=0x7fa7a404b590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=290b6c5a35c00d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7b17fa640 1 -- 172.21.15.150:0/1099756643 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a404b040 con 0x55df37c50590 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7b17fa640 1 -- 172.21.15.150:0/1099756643 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005670 con 0x55df37c50590 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7b17fa640 1 -- 172.21.15.150:0/1099756643 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40bd150 con 0x55df37c50590 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 -- 172.21.15.150:0/1099756643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 msgr2=0x55df37c5b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c5b8e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40edf30 tx=0x7fa7a404b590 comp rx=0 tx=0).stop 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 -- 172.21.15.150:0/1099756643 shutdown_connections 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c5be20 0x55df37c5e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c5b8e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1099756643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c4fc70 0x55df37c50050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 -- 172.21.15.150:0/1099756643 >> 172.21.15.150:0/1099756643 conn(0x55df37c4ac60 msgr2=0x55df37c4b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 -- 172.21.15.150:0/1099756643 shutdown_connections 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.070+0000 7fa7c4191740 1 -- 172.21.15.150:0/1099756643 wait complete. 2024-03-18T16:07:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 -- start start 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 Processor -- start 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c58f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c59450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55df37baeae0 con 0x55df37c5be20 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55df37b6ed70 con 0x55df37c50590 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55df37c5e750 con 0x55df37c4fc70 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7c4191740 10 client.0 _ll_register_callbacks cb 0x55df378c6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37386/0 (socket says 172.21.15.150:37386) 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1652907941 learned_addr learned my addr 172.21.15.150:0/1652907941 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c58f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c59450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.080 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 msgr2=0x55df37c58f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c58f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1652907941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 msgr2=0x55df37c59450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c59450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.071+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1652907941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df378c7210 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0137f0 tx=0x7fa7ac013cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b59239204fa6918 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac020060 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a4005670 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fa7ac004440 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa7ac0108b0 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 10 client.16032 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55df37993620 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005670 con 0x55df37c5be20 2024-03-18T16:07:34.081 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa7ac0233b0 con 0x55df37c5be20 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 10 client.16032 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 20 client.16032 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37c6bbf0 0x55df37c6dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 4 client.16032 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.072+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df37bbff80 con 0x55df37c6bbf0 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.073+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37c6bbf0 0x55df37c6dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.073+0000 7fa7a27fc640 10 client.16032 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.073+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37c6bbf0 0x55df37c6dfe0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=736cb171fc3a5f98 server_cookie=32f87a56e26e99bb in_seq=0 out_seq=0 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1652907941 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df37bbff80 con 0x55df37c6bbf0 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7a27fc640 10 client.16032 handle_command_reply: tid=0 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 client.16032 shutdown 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 msgr2=0x55df37c59990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0137f0 tx=0x7fa7ac013cb0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 shutdown_connections 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37c6bbf0 0x55df37c6dfe0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c59450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7a27fc640 0 client.16032 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1652907941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c58f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 >> 172.21.15.150:0/1652907941 conn(0x55df37c4ac60 msgr2=0x55df37c4c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 shutdown_connections 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 1 -- 172.21.15.150:0/1652907941 wait complete. 2024-03-18T16:07:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.074+0000 7fa7c4191740 20 client.16032 trim_cache size 0 max 16384 2024-03-18T16:07:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.089+0000 7fa7c4191740 1 Processor -- start 2024-03-18T16:07:34.094 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.089+0000 7fa7c4191740 1 -- start start 2024-03-18T16:07:34.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:33 smithi150 ceph-mon[31946]: pgmap v231: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c5b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1ada20 con 0x55df37c5be20 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1ab2f0 con 0x55df37c50590 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7c4191740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1a9160 con 0x55df37c4fc70 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:56950/0 (socket says 172.21.15.150:56950) 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1588191855 learned_addr learned my addr 172.21.15.150:0/1588191855 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c5b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1588191855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 msgr2=0x55df37c5b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c5b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1588191855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 msgr2=0x55df37c568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 -- 172.21.15.150:0/1588191855 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df37c6a790 con 0x55df37c50590 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.090+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40bcbf0 tx=0x7fa7a4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1adfc6264741f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7b0ff9640 1 -- 172.21.15.150:0/1588191855 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a404b040 con 0x55df37c50590 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7b0ff9640 1 -- 172.21.15.150:0/1588191855 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005670 con 0x55df37c50590 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7b0ff9640 1 -- 172.21.15.150:0/1588191855 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004c40 con 0x55df37c50590 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 -- 172.21.15.150:0/1588191855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 msgr2=0x55df37c47bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40bcbf0 tx=0x7fa7a4009680 comp rx=0 tx=0).stop 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 -- 172.21.15.150:0/1588191855 shutdown_connections 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c50590 0x55df37c47bb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1588191855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c4fc70 0x55df37c5b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 -- 172.21.15.150:0/1588191855 >> 172.21.15.150:0/1588191855 conn(0x55df37c4ac60 msgr2=0x55df37c4be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 -- 172.21.15.150:0/1588191855 shutdown_connections 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.091+0000 7fa7c4191740 1 -- 172.21.15.150:0/1588191855 wait complete. 2024-03-18T16:07:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 -- start start 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 Processor -- start 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c4fc70 0x55df37c58a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c58f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1a97b0 con 0x55df37c5be20 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1aac20 con 0x55df37c4fc70 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc1a9e80 con 0x55df37c50590 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7c4191740 10 client.0 _ll_register_callbacks cb 0x7fa7ac0133d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37406/0 (socket says 172.21.15.150:37406) 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1656076172 learned_addr learned my addr 172.21.15.150:0/1656076172 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c4fc70 0x55df37c58a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7ba7fc640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c58f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 msgr2=0x55df37c58f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c58f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1656076172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c4fc70 msgr2=0x55df37c58a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c4fc70 0x55df37c58a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 -- 172.21.15.150:0/1656076172 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df37cd0f80 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7bb7fe640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009660 tx=0x7fa7ac010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea63f7e4e8d6d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.092+0000 7fa7a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac023e50 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fa7ac0052b0 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a4005670 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa7ac0054b0 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 10 client.16038 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55df37cd2270 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005670 con 0x55df37c5be20 2024-03-18T16:07:34.102 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa7ac02b020 con 0x55df37c5be20 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 10 client.16038 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 20 client.16038 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37cd2990 0x55df37cd4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 4 client.16038 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df37bba2d0 con 0x55df37cd2990 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.093+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37cd2990 0x55df37cd4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.094+0000 7fa7baffd640 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37cd2990 0x55df37cd4d80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caad855ce16a8445 server_cookie=423e546b0b38727a in_seq=0 out_seq=0 2024-03-18T16:07:34.103 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.094+0000 7fa7a27fc640 10 client.16038 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.107+0000 7fa7a27fc640 1 -- 172.21.15.150:0/1656076172 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df37bba2d0 con 0x55df37cd2990 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.107+0000 7fa7a27fc640 10 client.16038 handle_command_reply: tid=0 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 client.16038 shutdown 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 msgr2=0x55df37c59490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009660 tx=0x7fa7ac010c80 comp rx=0 tx=0).stop 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 shutdown_connections 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55df37cd2990 0x55df37cd4d80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55df37c5be20 0x55df37c59490 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55df37c50590 0x55df37c58f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7a27fc640 0 client.16038 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 --2- 172.21.15.150:0/1656076172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df37c4fc70 0x55df37c58a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 >> 172.21.15.150:0/1656076172 conn(0x55df37c4ac60 msgr2=0x55df37c4be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 shutdown_connections 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 1 -- 172.21.15.150:0/1656076172 wait complete. 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.108+0000 7fa7c4191740 20 client.16038 trim_cache size 0 max 16384 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:07:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:07:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 msgr2=0x7fa7a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 0x7fa7a80a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40049d0 tx=0x7fa7b4009500 comp rx=0 tx=0).stop 2024-03-18T16:07:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 msgr2=0x7fa7bc1974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc1974a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4000c00 tx=0x7fa7a40042e0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 shutdown_connections 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fa7a80a2a10 0x7fa7a80a4ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa7bc105680 0x7fa7bc19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7bc1012f0 0x7fa7bc1974a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 --2- 172.21.15.150:0/980486650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7bc100930 0x7fa7bc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 >> 172.21.15.150:0/980486650 conn(0x7fa7bc077530 msgr2=0x7fa7bc0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 shutdown_connections 2024-03-18T16:07:34.114 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:34.109+0000 7fa7c14c6640 1 -- 172.21.15.150:0/980486650 wait complete. 2024-03-18T16:07:34.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 755949 10.82 MB/sec execute 259 sec latency 26.811 ms 2024-03-18T16:07:35.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 757385 10.81 MB/sec execute 260 sec latency 53.529 ms 2024-03-18T16:07:35.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:35 smithi191 ceph-mon[36784]: pgmap v232: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:35 smithi156 ceph-mon[35919]: pgmap v232: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:35 smithi150 ceph-mon[31946]: pgmap v232: 97 pgs: 97 active+clean; 468 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-18T16:07:36.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 759331 10.81 MB/sec execute 261 sec latency 39.449 ms 2024-03-18T16:07:37.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 761073 10.80 MB/sec execute 262 sec latency 13.034 ms 2024-03-18T16:07:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:37 smithi191 ceph-mon[36784]: pgmap v233: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 282 op/s 2024-03-18T16:07:38.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:37 smithi156 ceph-mon[35919]: pgmap v233: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 282 op/s 2024-03-18T16:07:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:37 smithi150 ceph-mon[31946]: pgmap v233: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.5 MiB/s wr, 282 op/s 2024-03-18T16:07:38.331 INFO:tasks.workunit.client.0.smithi150.stdout: 1 763242 10.80 MB/sec execute 263 sec latency 18.356 ms 2024-03-18T16:07:39.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 764812 10.80 MB/sec execute 264 sec latency 31.943 ms 2024-03-18T16:07:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:39 smithi191 ceph-mon[36784]: pgmap v234: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-18T16:07:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:39 smithi156 ceph-mon[35919]: pgmap v234: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-18T16:07:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:39 smithi150 ceph-mon[31946]: pgmap v234: 97 pgs: 97 active+clean; 480 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-18T16:07:40.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 766778 10.79 MB/sec execute 265 sec latency 47.236 ms 2024-03-18T16:07:41.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 768217 10.78 MB/sec execute 266 sec latency 61.485 ms 2024-03-18T16:07:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:41 smithi191 ceph-mon[36784]: pgmap v235: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-18T16:07:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:41 smithi156 ceph-mon[35919]: pgmap v235: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-18T16:07:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:41 smithi150 ceph-mon[31946]: pgmap v235: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 276 op/s 2024-03-18T16:07:42.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 770508 10.80 MB/sec execute 267 sec latency 18.325 ms 2024-03-18T16:07:43.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 771952 10.78 MB/sec execute 268 sec latency 44.218 ms 2024-03-18T16:07:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:43 smithi191 ceph-mon[36784]: pgmap v236: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:43 smithi156 ceph-mon[35919]: pgmap v236: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:43 smithi150 ceph-mon[31946]: pgmap v236: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:44.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 774152 10.79 MB/sec execute 269 sec latency 30.753 ms 2024-03-18T16:07:45.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 775606 10.77 MB/sec execute 270 sec latency 20.828 ms 2024-03-18T16:07:45.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:45 smithi191 ceph-mon[36784]: pgmap v237: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:45 smithi156 ceph-mon[35919]: pgmap v237: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:45 smithi150 ceph-mon[31946]: pgmap v237: 97 pgs: 97 active+clean; 486 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 180 op/s 2024-03-18T16:07:46.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 777589 10.78 MB/sec execute 271 sec latency 106.569 ms 2024-03-18T16:07:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 -- 172.21.15.150:0/3439058314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 msgr2=0x7f8104107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 --2- 172.21.15.150:0/3439058314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f8104107b80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8100009930 tx=0x7f810002bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 -- 172.21.15.150:0/3439058314 shutdown_connections 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 --2- 172.21.15.150:0/3439058314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104102150 0x7f810410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 --2- 172.21.15.150:0/3439058314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104101790 0x7f8104101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 --2- 172.21.15.150:0/3439058314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f8104107b80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 -- 172.21.15.150:0/3439058314 >> 172.21.15.150:0/3439058314 conn(0x7f81040fd2a0 msgr2=0x7f81040ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 -- 172.21.15.150:0/3439058314 shutdown_connections 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.198+0000 7f810bd37640 1 -- 172.21.15.150:0/3439058314 wait complete. 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 Processor -- start 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 -- start start 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104102150 0x7f8104198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f810419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8104111bb0 con 0x7f81041077a0 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8104111a30 con 0x7f8104101790 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810bd37640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8104111d30 con 0x7f8104102150 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f810a2d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f810419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:38682/0 (socket says 172.21.15.150:38682) 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f8109ad3640 1 -- 172.21.15.150:0/2283629650 learned_addr learned my addr 172.21.15.150:0/2283629650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.199+0000 7f81092d2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104102150 0x7f8104198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 -- 172.21.15.150:0/2283629650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104102150 msgr2=0x7f8104198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104102150 0x7f8104198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 -- 172.21.15.150:0/2283629650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 msgr2=0x7f8104198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 -- 172.21.15.150:0/2283629650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810419d020 con 0x7f81041077a0 2024-03-18T16:07:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f8109ad3640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810a2d4640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f810419c8f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00ec50 tx=0x7f80fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f80faffd640 1 -- 172.21.15.150:0/2283629650 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc003b70 con 0x7f81041077a0 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f80faffd640 1 -- 172.21.15.150:0/2283629650 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80fc003d10 con 0x7f81041077a0 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f810419d310 con 0x7f81041077a0 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f80faffd640 1 -- 172.21.15.150:0/2283629650 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc010890 con 0x7f81041077a0 2024-03-18T16:07:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.200+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81041a4ba0 con 0x7f81041077a0 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.202+0000 7f80faffd640 1 -- 172.21.15.150:0/2283629650 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f80fc003fc0 con 0x7f81041077a0 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 Processor -- start 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 -- start start 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f80faffd640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 0x7f80f00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110a50 0x562e7b1192e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 0x562e7b11bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562e7b074a40 con 0x562e7b119820 2024-03-18T16:07:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562e7b01d920 con 0x562e7b110a50 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810e9fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562e7b070ae0 con 0x562e7b110130 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810a2d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 0x562e7b11bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f81092d2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f8109ad3640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 0x7f80f00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f81092d2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:56200/0 (socket says 172.21.15.150:56200) 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f810a2d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 0x562e7b11bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42200/0 (socket says 172.21.15.150:42200) 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f81092d2640 1 -- 172.21.15.150:0/82220357 learned_addr learned my addr 172.21.15.150:0/82220357 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f81092d2640 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110a50 0x562e7b1192e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.203+0000 7f80faffd640 1 -- 172.21.15.150:0/2283629650 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f80fc0cf390 con 0x7f81041077a0 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 -- 172.21.15.150:0/82220357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110a50 msgr2=0x562e7b1192e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110a50 0x562e7b1192e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 -- 172.21.15.150:0/82220357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 msgr2=0x562e7b11bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 0x562e7b11bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 -- 172.21.15.150:0/82220357 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e7b035460 con 0x562e7b110130 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f81092d2640 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f80f40117e0 tx=0x7f80f40097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64ffcfb9193f6652 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f8109ad3640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 0x7f80f00a4ed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8100000c00 tx=0x7f81000058d0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f80effff640 1 -- 172.21.15.150:0/82220357 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4004030 con 0x562e7b110130 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f80effff640 1 -- 172.21.15.150:0/82220357 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f400fca0 con 0x562e7b110130 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.204+0000 7f80effff640 1 -- 172.21.15.150:0/82220357 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f40038e0 con 0x562e7b110130 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- 172.21.15.150:0/82220357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 msgr2=0x562e7b110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f80f40117e0 tx=0x7f80f40097d0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- 172.21.15.150:0/82220357 shutdown_connections 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b119820 0x562e7b11bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110a50 0x562e7b1192e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 --2- 172.21.15.150:0/82220357 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b110510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- 172.21.15.150:0/82220357 >> 172.21.15.150:0/82220357 conn(0x562e7b10aa50 msgr2=0x562e7b10ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- 172.21.15.150:0/82220357 shutdown_connections 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- 172.21.15.150:0/82220357 wait complete. 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 -- start start 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.205+0000 7f810e9fc740 1 Processor -- start 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b116880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b119820 0x562e7b117300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562e7b11c450 con 0x562e7b110a50 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562e7b030d70 con 0x562e7b119820 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562e7b11c5d0 con 0x562e7b110130 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 10 client.0 _ll_register_callbacks cb 0x562e7ad88a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42204/0 (socket says 172.21.15.150:42204) 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810a2d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b119820 0x562e7b117300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b116880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 -- 172.21.15.150:0/3310405654 learned_addr learned my addr 172.21.15.150:0/3310405654 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 -- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 msgr2=0x562e7b116880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b116880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 -- 172.21.15.150:0/3310405654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b119820 msgr2=0x562e7b117300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b119820 0x562e7b117300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 -- 172.21.15.150:0/3310405654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e7ad89210 con 0x562e7b110a50 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f81092d2640 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f80f4004500 tx=0x7f80f4009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4867cca50bbae790 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f80ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4005260 con 0x562e7b110a50 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f80f4005400 con 0x562e7b110a50 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f400fca0 con 0x562e7b110a50 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f80f4005600 con 0x562e7b110a50 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 10 client.16050 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.206+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562e7ae55620 con 0x562e7b110a50 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f400fca0 con 0x562e7b110a50 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f80f4026070 con 0x562e7b110a50 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 10 client.16050 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 20 client.16050 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b129860 0x562e7b12bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 4 client.16050 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e7b081f80 con 0x562e7b129860 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f8109ad3640 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b129860 0x562e7b12bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f80ed7fa640 10 client.16050 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.207+0000 7f8109ad3640 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b129860 0x562e7b12bc50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0b9ac9198fe1f9e server_cookie=f3879a4049256415 in_seq=0 out_seq=0 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f80ed7fa640 1 -- 172.21.15.150:0/3310405654 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e7b081f80 con 0x562e7b129860 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f80ed7fa640 10 client.16050 handle_command_reply: tid=0 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 client.16050 shutdown 2024-03-18T16:07:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 msgr2=0x562e7b116dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f80f4004500 tx=0x7f80f4009fa0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 shutdown_connections 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b129860 0x562e7b12bc50 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b119820 0x562e7b117300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110a50 0x562e7b116dc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f80ed7fa640 0 client.16050 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3310405654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110130 0x562e7b116880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.208+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 >> 172.21.15.150:0/3310405654 conn(0x562e7b10aa50 msgr2=0x562e7b10c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.209+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 shutdown_connections 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.209+0000 7f810e9fc740 1 -- 172.21.15.150:0/3310405654 wait complete. 2024-03-18T16:07:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.209+0000 7f810e9fc740 20 client.16050 trim_cache size 0 max 16384 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.223+0000 7f810e9fc740 1 Processor -- start 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.223+0000 7f810e9fc740 1 -- start start 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.223+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b10aa50 0x562e7b118fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.223+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.223+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b1251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f810e9fc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81041afce0 con 0x562e7b110130 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f810e9fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ad5b0 con 0x562e7b10aa50 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f810e9fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ab420 con 0x562e7b110a50 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42208/0 (socket says 172.21.15.150:42208) 2024-03-18T16:07:47.227 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 -- 172.21.15.150:0/3216270009 learned_addr learned my addr 172.21.15.150:0/3216270009 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b10aa50 0x562e7b118fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f810a2d4640 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b1251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 -- 172.21.15.150:0/3216270009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 msgr2=0x562e7b1251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b1251f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 -- 172.21.15.150:0/3216270009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b10aa50 msgr2=0x562e7b118fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b10aa50 0x562e7b118fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 -- 172.21.15.150:0/3216270009 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e7b115830 con 0x562e7b110130 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f81092d2640 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80f40057c0 tx=0x7f80f40115d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b9398ab7fe25d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f80ef7fe640 1 -- 172.21.15.150:0/3216270009 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4028440 con 0x562e7b110130 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f80ef7fe640 1 -- 172.21.15.150:0/3216270009 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f4009c60 con 0x562e7b110130 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.224+0000 7f80ef7fe640 1 -- 172.21.15.150:0/3216270009 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4004b60 con 0x562e7b110130 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- 172.21.15.150:0/3216270009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 msgr2=0x562e7b105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80f40057c0 tx=0x7f80f40115d0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- 172.21.15.150:0/3216270009 shutdown_connections 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b1251f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b110130 0x562e7b105a40 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 --2- 172.21.15.150:0/3216270009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b10aa50 0x562e7b118fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- 172.21.15.150:0/3216270009 >> 172.21.15.150:0/3216270009 conn(0x562e7b12bfc0 msgr2=0x562e7b109bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- 172.21.15.150:0/3216270009 shutdown_connections 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- 172.21.15.150:0/3216270009 wait complete. 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 -- start start 2024-03-18T16:07:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.225+0000 7f810e9fc740 1 Processor -- start 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110130 0x562e7b11dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b18ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81041aba70 con 0x562e7b10aa50 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f81041acee0 con 0x562e7b110130 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f81041ac140 con 0x562e7b110a50 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 10 client.0 _ll_register_callbacks cb 0x7f80fc097650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:07:47.232 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42218/0 (socket says 172.21.15.150:42218) 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 -- 172.21.15.150:0/1896304269 learned_addr learned my addr 172.21.15.150:0/1896304269 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810a2d4640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b18ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f81092d2640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110130 0x562e7b11dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 -- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 msgr2=0x562e7b18ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b18ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 -- 172.21.15.150:0/1896304269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110130 msgr2=0x562e7b11dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110130 0x562e7b11dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 -- 172.21.15.150:0/1896304269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e7b18ef70 con 0x562e7b10aa50 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f8109ad3640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f810002e020 tx=0x7f810002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8822fdd74204bc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f80ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100056070 con 0x562e7b10aa50 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7f8100052d00 con 0x562e7b10aa50 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80f4009c60 con 0x562e7b10aa50 2024-03-18T16:07:47.233 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8100004040 con 0x562e7b10aa50 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.226+0000 7f810e9fc740 10 client.16056 fetch_fsmap learned FSMap version 31 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562e7b190280 con 0x562e7b10aa50 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f4009c60 con 0x562e7b10aa50 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f8100004f80 con 0x562e7b10aa50 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 10 client.16056 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 20 client.16056 populate_metadata read hostname 'smithi150' 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b119820 0x562e7b192990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 4 client.16056 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e7b07c2d0 con 0x562e7b119820 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f81092d2640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b119820 0x562e7b192990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f80ed7fa640 10 client.16056 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:47.234 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.227+0000 7f81092d2640 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b119820 0x562e7b192990 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acaa5ac415b346f5 server_cookie=c9a5f5bf36bd258a in_seq=0 out_seq=0 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.238+0000 7f80ed7fa640 1 -- 172.21.15.150:0/1896304269 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e7b07c2d0 con 0x562e7b119820 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.238+0000 7f80ed7fa640 10 client.16056 handle_command_reply: tid=0 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.238+0000 7f810e9fc740 1 client.16056 shutdown 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 msgr2=0x562e7b11d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f810002e020 tx=0x7f810002bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 shutdown_connections 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562e7b119820 0x562e7b192990 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562e7b110a50 0x562e7b18ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f80ed7fa640 0 client.16056 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e7b110130 0x562e7b11dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 --2- 172.21.15.150:0/1896304269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562e7b10aa50 0x562e7b11d7e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 >> 172.21.15.150:0/1896304269 conn(0x562e7b12bfc0 msgr2=0x562e7b10bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.241 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 shutdown_connections 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 1 -- 172.21.15.150:0/1896304269 wait complete. 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810e9fc740 20 client.16056 trim_cache size 0 max 16384 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:07:47.242 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 msgr2=0x7f80f00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 0x7f80f00a4ed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8100000c00 tx=0x7f81000058d0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 msgr2=0x7f810419c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f810419c8f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00ec50 tx=0x7f80fc00c6a0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 shutdown_connections 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f80f00a2a10 0x7f80f00a4ed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81041077a0 0x7f810419c8f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.243 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8104102150 0x7f8104198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 --2- 172.21.15.150:0/2283629650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104101790 0x7f8104198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.239+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 >> 172.21.15.150:0/2283629650 conn(0x7f81040fd2a0 msgr2=0x7f81040fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.240+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 shutdown_connections 2024-03-18T16:07:47.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:07:47.240+0000 7f810bd37640 1 -- 172.21.15.150:0/2283629650 wait complete. 2024-03-18T16:07:47.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 779083 10.76 MB/sec execute 272 sec latency 26.427 ms 2024-03-18T16:07:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:47 smithi191 ceph-mon[36784]: pgmap v238: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-03-18T16:07:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:47 smithi156 ceph-mon[35919]: pgmap v238: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-03-18T16:07:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:47 smithi150 ceph-mon[31946]: pgmap v238: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 272 op/s 2024-03-18T16:07:48.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 781346 10.77 MB/sec execute 273 sec latency 23.509 ms 2024-03-18T16:07:49.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 782669 10.75 MB/sec execute 274 sec latency 25.878 ms 2024-03-18T16:07:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:49 smithi191 ceph-mon[36784]: pgmap v239: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:07:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:49 smithi156 ceph-mon[35919]: pgmap v239: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:07:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:49 smithi150 ceph-mon[31946]: pgmap v239: 97 pgs: 97 active+clean; 464 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:07:50.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 784905 10.76 MB/sec execute 275 sec latency 28.156 ms 2024-03-18T16:07:51.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 786177 10.75 MB/sec execute 276 sec latency 76.711 ms 2024-03-18T16:07:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:51 smithi191 ceph-mon[36784]: pgmap v240: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-03-18T16:07:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:51 smithi156 ceph-mon[35919]: pgmap v240: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-03-18T16:07:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:51 smithi150 ceph-mon[31946]: pgmap v240: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 266 op/s 2024-03-18T16:07:52.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 788316 10.75 MB/sec execute 277 sec latency 21.533 ms 2024-03-18T16:07:53.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 789748 10.74 MB/sec execute 278 sec latency 22.510 ms 2024-03-18T16:07:53.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:53 smithi191 ceph-mon[36784]: pgmap v241: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:53 smithi156 ceph-mon[35919]: pgmap v241: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:53 smithi150 ceph-mon[31946]: pgmap v241: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:54.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 791846 10.74 MB/sec execute 279 sec latency 36.649 ms 2024-03-18T16:07:55.332 INFO:tasks.workunit.client.0.smithi150.stdout: 1 793334 10.73 MB/sec execute 280 sec latency 17.251 ms 2024-03-18T16:07:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:55 smithi191 ceph-mon[36784]: pgmap v242: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:55 smithi156 ceph-mon[35919]: pgmap v242: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:55 smithi150 ceph-mon[31946]: pgmap v242: 97 pgs: 97 active+clean; 464 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 177 op/s 2024-03-18T16:07:56.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 795449 10.74 MB/sec execute 281 sec latency 57.294 ms 2024-03-18T16:07:57.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 796916 10.72 MB/sec execute 282 sec latency 20.005 ms 2024-03-18T16:07:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:57 smithi191 ceph-mon[36784]: pgmap v243: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-03-18T16:07:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:57 smithi156 ceph-mon[35919]: pgmap v243: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-03-18T16:07:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:57 smithi150 ceph-mon[31946]: pgmap v243: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.7 MiB/s wr, 265 op/s 2024-03-18T16:07:58.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 799143 10.73 MB/sec execute 283 sec latency 16.849 ms 2024-03-18T16:07:59.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 800637 10.71 MB/sec execute 284 sec latency 33.381 ms 2024-03-18T16:07:59.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:07:59 smithi191 ceph-mon[36784]: pgmap v244: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-03-18T16:08:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:07:59 smithi156 ceph-mon[35919]: pgmap v244: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-03-18T16:08:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:07:59 smithi150 ceph-mon[31946]: pgmap v244: 97 pgs: 97 active+clean; 471 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.330+0000 7fba390d1640 1 -- 172.21.15.150:0/3323112872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 msgr2=0x7fba3410e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.330+0000 7fba390d1640 1 --2- 172.21.15.150:0/3323112872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba3410e6a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009930 tx=0x7fba2c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 -- 172.21.15.150:0/3323112872 shutdown_connections 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 --2- 172.21.15.150:0/3323112872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba3410e6a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 --2- 172.21.15.150:0/3323112872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 0x7fba34101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 --2- 172.21.15.150:0/3323112872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 0x7fba34107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 -- 172.21.15.150:0/3323112872 >> 172.21.15.150:0/3323112872 conn(0x7fba340fd2a0 msgr2=0x7fba340ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 -- 172.21.15.150:0/3323112872 shutdown_connections 2024-03-18T16:08:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 -- 172.21.15.150:0/3323112872 wait complete. 2024-03-18T16:08:00.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 802751 10.72 MB/sec execute 285 sec latency 18.225 ms 2024-03-18T16:08:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.331+0000 7fba390d1640 1 Processor -- start 2024-03-18T16:08:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 -- start start 2024-03-18T16:08:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 0x7fba34073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 0x7fba34074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba34111bf0 con 0x7fba34102190 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba34111a70 con 0x7fba341077e0 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba390d1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba34111d70 con 0x7fba341017d0 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48148/0 (socket says 172.21.15.150:48148) 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 -- 172.21.15.150:0/1773730859 learned_addr learned my addr 172.21.15.150:0/1773730859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 0x7fba34073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3359e640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 0x7fba34074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 -- 172.21.15.150:0/1773730859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 msgr2=0x7fba34073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 0x7fba34073ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 -- 172.21.15.150:0/1773730859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 msgr2=0x7fba34074560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 0x7fba34074560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 -- 172.21.15.150:0/1773730859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba341a6490 con 0x7fba34102190 2024-03-18T16:08:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.332+0000 7fba3259c640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00e9f0 tx=0x7fba1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.333+0000 7fba2bfff640 1 -- 172.21.15.150:0/1773730859 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00cd80 con 0x7fba34102190 2024-03-18T16:08:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.333+0000 7fba2bfff640 1 -- 172.21.15.150:0/1773730859 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba1c005500 con 0x7fba34102190 2024-03-18T16:08:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.333+0000 7fba2bfff640 1 -- 172.21.15.150:0/1773730859 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c010690 con 0x7fba34102190 2024-03-18T16:08:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.333+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba341a6780 con 0x7fba34102190 2024-03-18T16:08:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.333+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba341a6cd0 con 0x7fba34102190 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.335+0000 7fba2bfff640 1 -- 172.21.15.150:0/1773730859 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fba1c0046c0 con 0x7fba34102190 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.335+0000 7fba3bd98740 1 Processor -- start 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.335+0000 7fba3bd98740 1 -- start start 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eeca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eeccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559c0ee25a40 con 0x559c0eec1a50 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c0edce920 con 0x559c0eec1130 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3bd98740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559c0ee21ae0 con 0x559c0eeca820 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba2bfff640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 0x7fba200a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eeca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:47014/0 (socket says 172.21.15.150:47014) 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3359e640 1 -- 172.21.15.150:0/2845742591 learned_addr learned my addr 172.21.15.150:0/2845742591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eeccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3359e640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 0x7fba200a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba2bfff640 1 -- 172.21.15.150:0/1773730859 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fba1c0cf1f0 con 0x7fba34102190 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 -- 172.21.15.150:0/2845742591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 msgr2=0x559c0eeccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eeccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 -- 172.21.15.150:0/2845742591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 msgr2=0x559c0eec1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 -- 172.21.15.150:0/2845742591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c0ede6460 con 0x559c0eec1a50 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba3359e640 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.336+0000 7fba32d9d640 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eeca2e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fba24009fa0 tx=0x7fba240118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3d85c7c543109 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba3359e640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 0x7fba200a4f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba2c006010 tx=0x7fba2c0058d0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba297fa640 1 -- 172.21.15.150:0/2845742591 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba24003c30 con 0x559c0eec1a50 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba297fa640 1 -- 172.21.15.150:0/2845742591 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba24003dd0 con 0x559c0eec1a50 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba297fa640 1 -- 172.21.15.150:0/2845742591 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2400fac0 con 0x559c0eec1a50 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba3bd98740 1 -- 172.21.15.150:0/2845742591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 msgr2=0x559c0eeca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.337+0000 7fba3bd98740 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eeca2e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fba24009fa0 tx=0x7fba240118d0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- 172.21.15.150:0/2845742591 shutdown_connections 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eeccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eeca2e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- 172.21.15.150:0/2845742591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- 172.21.15.150:0/2845742591 >> 172.21.15.150:0/2845742591 conn(0x559c0eebba50 msgr2=0x559c0eebbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- 172.21.15.150:0/2845742591 shutdown_connections 2024-03-18T16:08:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- 172.21.15.150:0/2845742591 wait complete. 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- start start 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 Processor -- start 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eec7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 0x559c0eec82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559c0eecd450 con 0x559c0eec1a50 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559c0ede1d70 con 0x559c0eeca820 2024-03-18T16:08:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x559c0eecd5d0 con 0x559c0eec1130 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3bd98740 10 client.0 _ll_register_callbacks cb 0x559c0eb39a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48166/0 (socket says 172.21.15.150:48166) 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3259c640 1 -- 172.21.15.150:0/1815639496 learned_addr learned my addr 172.21.15.150:0/1815639496 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba3359e640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 0x559c0eec82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.338+0000 7fba32d9d640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eec7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 -- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 msgr2=0x559c0eec7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eec7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 -- 172.21.15.150:0/1815639496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 msgr2=0x559c0eec7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 -- 172.21.15.150:0/1815639496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c0eb3a210 con 0x559c0eeca820 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3259c640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3359e640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 0x559c0eec82b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba2c005c40 tx=0x7fba2c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697a587198a159bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:08:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c004f10 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fba2c0050b0 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba24003dd0 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba2c0052b0 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3bd98740 10 client.25832 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559c0ec06620 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.339+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba24003dd0 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fba2c04c460 con 0x559c0eeca820 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 10 client.25832 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 10 client.25832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 20 client.25832 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0eeda980 0x559c0eedcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 4 client.25832 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c0ee32f80 con 0x559c0eeda980 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba32d9d640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0eeda980 0x559c0eedcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba1a7fc640 10 client.25832 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.340+0000 7fba32d9d640 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0eeda980 0x559c0eedcd90 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb3e781317c773 server_cookie=8ee866b4bbbe6bce in_seq=0 out_seq=0 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba1a7fc640 1 -- 172.21.15.150:0/1815639496 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c0ee32f80 con 0x559c0eeda980 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba1a7fc640 10 client.25832 handle_command_reply: tid=0 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 client.25832 shutdown 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 msgr2=0x559c0eec82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 0x559c0eec82b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba2c005c40 tx=0x7fba2c009260 comp rx=0 tx=0).stop 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 shutdown_connections 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0eeda980 0x559c0eedcd90 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eeca820 0x559c0eec82b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba1a7fc640 0 client.25832 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eec7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1815639496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eec7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 >> 172.21.15.150:0/1815639496 conn(0x559c0eebba50 msgr2=0x559c0eebd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.341+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 shutdown_connections 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.342+0000 7fba3bd98740 1 -- 172.21.15.150:0/1815639496 wait complete. 2024-03-18T16:08:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.342+0000 7fba3bd98740 20 client.25832 trim_cache size 0 max 16384 2024-03-18T16:08:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.356+0000 7fba3bd98740 1 Processor -- start 2024-03-18T16:08:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.356+0000 7fba3bd98740 1 -- start start 2024-03-18T16:08:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eee26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba341b1f20 con 0x559c0eec1a50 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba341af7f0 con 0x559c0eec1130 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3bd98740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba341ad660 con 0x559c0eeca820 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48178/0 (socket says 172.21.15.150:48178) 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 -- 172.21.15.150:0/1689282011 learned_addr learned my addr 172.21.15.150:0/1689282011 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3359e640 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eee26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba32d9d640 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 -- 172.21.15.150:0/1689282011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 msgr2=0x559c0eee26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eee26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 -- 172.21.15.150:0/1689282011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 msgr2=0x559c0eec9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 -- 172.21.15.150:0/1689282011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c0eed6730 con 0x559c0eec1a50 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba3259c640 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fba1c097cb0 tx=0x7fba1c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c614aaf45577a233 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.360 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba28ff9640 1 -- 172.21.15.150:0/1689282011 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c09b1b0 con 0x559c0eec1a50 2024-03-18T16:08:00.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba28ff9640 1 -- 172.21.15.150:0/1689282011 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba1c09bf50 con 0x559c0eec1a50 2024-03-18T16:08:00.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.357+0000 7fba28ff9640 1 -- 172.21.15.150:0/1689282011 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1c0a3b60 con 0x559c0eec1a50 2024-03-18T16:08:00.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 -- 172.21.15.150:0/1689282011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 msgr2=0x559c0eed61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.361 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fba1c097cb0 tx=0x7fba1c01a040 comp rx=0 tx=0).stop 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 -- 172.21.15.150:0/1689282011 shutdown_connections 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eeca820 0x559c0eee26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eec1a50 0x559c0eed61f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 --2- 172.21.15.150:0/1689282011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1130 0x559c0eec9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 -- 172.21.15.150:0/1689282011 >> 172.21.15.150:0/1689282011 conn(0x559c0eebba50 msgr2=0x559c0eec6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 -- 172.21.15.150:0/1689282011 shutdown_connections 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.358+0000 7fba3bd98740 1 -- 172.21.15.150:0/1689282011 wait complete. 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 -- start start 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 Processor -- start 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eed2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1a50 0x559c0eec40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba341adcb0 con 0x559c0eeca820 2024-03-18T16:08:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fba341af120 con 0x559c0eec1a50 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fba341ae380 con 0x559c0eec1130 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3bd98740 10 client.0 _ll_register_callbacks cb 0x7fba2c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48182/0 (socket says 172.21.15.150:48182) 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 -- 172.21.15.150:0/3530195633 learned_addr learned my addr 172.21.15.150:0/3530195633 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3259c640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1a50 0x559c0eec40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba32d9d640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eed2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 -- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 msgr2=0x559c0eed2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eed2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 -- 172.21.15.150:0/3530195633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1a50 msgr2=0x559c0eec40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1a50 0x559c0eec40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.359+0000 7fba3359e640 1 -- 172.21.15.150:0/3530195633 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c0eec4cc0 con 0x559c0eeca820 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3359e640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009230 tx=0x7fba2c04cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb94d97ce454a2bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c063060 con 0x559c0eeca820 2024-03-18T16:08:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8527+0+0 (secure 0 0 0) 0x7fba2c005650 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba1c09bf50 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba2c02ca30 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 10 client.16074 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559c0ef40ba0 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba1c09bf50 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fba2c003c60 con 0x559c0eeca820 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 10 client.16074 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 20 client.16074 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0ef412c0 0x559c0ef436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 4 client.16074 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c0ee2d2d0 con 0x559c0ef412c0 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.360+0000 7fba32d9d640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0ef412c0 0x559c0ef436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.361+0000 7fba32d9d640 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0ef412c0 0x559c0ef436b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdcc3e7301231ce server_cookie=a186103b4b1194ed in_seq=0 out_seq=0 2024-03-18T16:08:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.361+0000 7fba1a7fc640 10 client.16074 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.371+0000 7fba1a7fc640 1 -- 172.21.15.150:0/3530195633 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c0ee2d2d0 con 0x559c0ef412c0 2024-03-18T16:08:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.371+0000 7fba1a7fc640 10 client.16074 handle_command_reply: tid=0 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.371+0000 7fba3bd98740 1 client.16074 shutdown 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.371+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 msgr2=0x559c0eec4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.371+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009230 tx=0x7fba2c04cd20 comp rx=0 tx=0).stop 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 shutdown_connections 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559c0ef412c0 0x559c0ef436b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c0eeca820 0x559c0eec4600 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c0eec1a50 0x559c0eec40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 --2- 172.21.15.150:0/3530195633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559c0eec1130 0x559c0eed2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba1a7fc640 0 client.16074 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 >> 172.21.15.150:0/3530195633 conn(0x559c0eebba50 msgr2=0x559c0eec8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 shutdown_connections 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 1 -- 172.21.15.150:0/3530195633 wait complete. 2024-03-18T16:08:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba3bd98740 20 client.16074 trim_cache size 0 max 16384 2024-03-18T16:08:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:08:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:08:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:08:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 msgr2=0x7fba200a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 0x7fba200a4f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba2c006010 tx=0x7fba2c0058d0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 msgr2=0x7fba34074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00e9f0 tx=0x7fba1c00edf0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 shutdown_connections 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fba200a2ac0 0x7fba200a4f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba341077e0 0x7fba34074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba34102190 0x7fba34074020 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 --2- 172.21.15.150:0/1773730859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba341017d0 0x7fba34073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 >> 172.21.15.150:0/1773730859 conn(0x7fba340fd2a0 msgr2=0x7fba340fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.372+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 shutdown_connections 2024-03-18T16:08:00.376 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:00.373+0000 7fba390d1640 1 -- 172.21.15.150:0/1773730859 wait complete. 2024-03-18T16:08:01.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 804030 10.70 MB/sec execute 286 sec latency 96.357 ms 2024-03-18T16:08:01.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:01 smithi191 ceph-mon[36784]: pgmap v245: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-03-18T16:08:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:01 smithi156 ceph-mon[35919]: pgmap v245: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-03-18T16:08:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:01 smithi150 ceph-mon[31946]: pgmap v245: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 269 op/s 2024-03-18T16:08:02.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 806100 10.71 MB/sec execute 287 sec latency 34.615 ms 2024-03-18T16:08:03.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 807463 10.70 MB/sec execute 288 sec latency 21.244 ms 2024-03-18T16:08:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:03 smithi191 ceph-mon[36784]: pgmap v246: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-03-18T16:08:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:03 smithi191 ceph-mon[36784]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:08:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:03 smithi156 ceph-mon[35919]: pgmap v246: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-03-18T16:08:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:03 smithi156 ceph-mon[35919]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:08:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:03 smithi150 ceph-mon[31946]: pgmap v246: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 184 op/s 2024-03-18T16:08:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:03 smithi150 ceph-mon[31946]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:08:04.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 809329 10.69 MB/sec execute 289 sec latency 73.116 ms 2024-03-18T16:08:05.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 810752 10.68 MB/sec execute 290 sec latency 47.670 ms 2024-03-18T16:08:05.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:05 smithi191 ceph-mon[36784]: pgmap v248: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-03-18T16:08:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:05 smithi156 ceph-mon[35919]: pgmap v248: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-03-18T16:08:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:05 smithi150 ceph-mon[31946]: pgmap v248: 97 pgs: 97 active+clean; 479 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 221 op/s 2024-03-18T16:08:06.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 812676 10.68 MB/sec execute 291 sec latency 115.396 ms 2024-03-18T16:08:07.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 814187 10.67 MB/sec execute 292 sec latency 48.691 ms 2024-03-18T16:08:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:07 smithi191 ceph-mon[36784]: pgmap v249: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:07 smithi156 ceph-mon[35919]: pgmap v249: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:07 smithi150 ceph-mon[31946]: pgmap v249: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:08.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 816251 10.67 MB/sec execute 293 sec latency 18.356 ms 2024-03-18T16:08:09.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 818046 10.67 MB/sec execute 294 sec latency 20.671 ms 2024-03-18T16:08:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:09 smithi191 ceph-mon[36784]: pgmap v250: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:09 smithi191 ceph-mon[36784]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:08:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:09 smithi156 ceph-mon[35919]: pgmap v250: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:09 smithi156 ceph-mon[35919]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:08:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:09 smithi150 ceph-mon[31946]: pgmap v250: 97 pgs: 97 active+clean; 486 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-03-18T16:08:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:09 smithi150 ceph-mon[31946]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:08:10.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 820199 10.68 MB/sec execute 295 sec latency 20.242 ms 2024-03-18T16:08:11.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 821520 10.67 MB/sec execute 296 sec latency 60.199 ms 2024-03-18T16:08:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:11 smithi191 ceph-mon[36784]: pgmap v252: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-03-18T16:08:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:11 smithi156 ceph-mon[35919]: pgmap v252: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-03-18T16:08:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:11 smithi150 ceph-mon[31946]: pgmap v252: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 283 op/s 2024-03-18T16:08:12.333 INFO:tasks.workunit.client.0.smithi150.stdout: 1 823342 10.66 MB/sec execute 297 sec latency 47.528 ms 2024-03-18T16:08:13.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 825084 10.66 MB/sec execute 298 sec latency 19.676 ms 2024-03-18T16:08:13.463 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 -- 172.21.15.150:0/3519246868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af4101790 msgr2=0x7f7af4101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/3519246868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af4101790 0x7f7af4101c10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009930 tx=0x7f7adc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 -- 172.21.15.150:0/3519246868 shutdown_connections 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/3519246868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/3519246868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af4101790 0x7f7af4101c10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/3519246868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af41077a0 0x7f7af4107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.461+0000 7f7af9f3b640 1 -- 172.21.15.150:0/3519246868 >> 172.21.15.150:0/3519246868 conn(0x7f7af40fd2a0 msgr2=0x7f7af40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- 172.21.15.150:0/3519246868 shutdown_connections 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- 172.21.15.150:0/3519246868 wait complete. 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 Processor -- start 2024-03-18T16:08:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- start start 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af4101790 0x7f7af4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7af4111bb0 con 0x7f7af41077a0 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af4111a30 con 0x7f7af4102150 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.462+0000 7f7af9f3b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7af4111d30 con 0x7f7af4101790 2024-03-18T16:08:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af37fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af4101790 0x7f7af4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56544/0 (socket says 172.21.15.150:56544) 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 -- 172.21.15.150:0/2673449074 learned_addr learned my addr 172.21.15.150:0/2673449074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 -- 172.21.15.150:0/2673449074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af4101790 msgr2=0x7f7af4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af4101790 0x7f7af4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 -- 172.21.15.150:0/2673449074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 msgr2=0x7f7af41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 -- 172.21.15.150:0/2673449074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af419d010 con 0x7f7af41077a0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af3fff640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00efd0 tx=0x7f7aec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af17fa640 1 -- 172.21.15.150:0/2673449074 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec003b70 con 0x7f7af41077a0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af419d300 con 0x7f7af41077a0 2024-03-18T16:08:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af17fa640 1 -- 172.21.15.150:0/2673449074 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aec003d10 con 0x7f7af41077a0 2024-03-18T16:08:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af17fa640 1 -- 172.21.15.150:0/2673449074 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0109f0 con 0x7f7af41077a0 2024-03-18T16:08:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.463+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af41a4b90 con 0x7f7af41077a0 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7af17fa640 1 -- 172.21.15.150:0/2673449074 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f7aec004020 con 0x7f7af41077a0 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 Processor -- start 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 -- start start 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396ba410 0x560b396c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396ca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b3961ea40 con 0x560b396ba410 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b395c7920 con 0x560b396c7c30 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.466+0000 7f7afcbff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b3961aae0 con 0x560b396b9af0 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7af37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396ba410 0x560b396c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:47820/0 (socket says 172.21.15.150:47820) 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 -- 172.21.15.150:0/899496950 learned_addr learned my addr 172.21.15.150:0/899496950 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7af17fa640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 0x7f7ae00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7af3fff640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 0x7f7ae00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 -- 172.21.15.150:0/899496950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 msgr2=0x560b396ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 -- 172.21.15.150:0/899496950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396ba410 msgr2=0x560b396c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396ba410 0x560b396c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 -- 172.21.15.150:0/899496950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b395df460 con 0x560b396b9af0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7af17fa640 1 -- 172.21.15.150:0/2673449074 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f7aec0cf270 con 0x7f7af41077a0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7adc005c40 tx=0x7f7adc005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=822e6bd8629d0583 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7af3fff640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 0x7f7ae00a4ed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7aec005c40 tx=0x7f7aec009f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7ae9ffb640 1 -- 172.21.15.150:0/899496950 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc02ca80 con 0x560b396b9af0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7ae9ffb640 1 -- 172.21.15.150:0/899496950 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc02cc20 con 0x560b396b9af0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.467+0000 7f7ae9ffb640 1 -- 172.21.15.150:0/899496950 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc005520 con 0x560b396b9af0 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- 172.21.15.150:0/899496950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 msgr2=0x560b396b9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7adc005c40 tx=0x7f7adc005b70 comp rx=0 tx=0).stop 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- 172.21.15.150:0/899496950 shutdown_connections 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396ba410 0x560b396c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 --2- 172.21.15.150:0/899496950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396b9ed0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- 172.21.15.150:0/899496950 >> 172.21.15.150:0/899496950 conn(0x560b396b4a50 msgr2=0x560b396b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- 172.21.15.150:0/899496950 shutdown_connections 2024-03-18T16:08:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- 172.21.15.150:0/899496950 wait complete. 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.468+0000 7f7afcbff740 1 -- start start 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 Processor -- start 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396c5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560b396ca7a0 con 0x560b396b9af0 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560b395dad70 con 0x560b396c7c30 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560b396ca920 con 0x560b396ba410 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7afcbff740 10 client.0 _ll_register_callbacks cb 0x560b39332a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7af37fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56562/0 (socket says 172.21.15.150:56562) 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7af37fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396c5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:47834/0 (socket says 172.21.15.150:47834) 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 -- 172.21.15.150:0/540591430 learned_addr learned my addr 172.21.15.150:0/540591430 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 -- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 msgr2=0x560b396c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 -- 172.21.15.150:0/540591430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 msgr2=0x560b396c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 -- 172.21.15.150:0/540591430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b39333210 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02cfd0 tx=0x7f7adc005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c34ec2b3c2f1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.469+0000 7f7ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc004030 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f7adc0041d0 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc02cc20 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7adc0043d0 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 10 client.16086 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560b393ff620 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc02cc20 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f7adc0345c0 con 0x560b396b9af0 2024-03-18T16:08:13.478 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 10 client.16086 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 20 client.16086 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b396d7b60 0x560b396d9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 4 client.16086 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b3962bf80 con 0x560b396d7b60 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.470+0000 7f7af37fe640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b396d7b60 0x560b396d9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.471+0000 7f7af37fe640 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b396d7b60 0x560b396d9f50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6fa70d30c4adcc server_cookie=34a36d5ed835f3c1 in_seq=0 out_seq=0 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.471+0000 7f7ad2ffd640 10 client.16086 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/540591430 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b3962bf80 con 0x560b396d7b60 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7ad2ffd640 10 client.16086 handle_command_reply: tid=0 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 client.16086 shutdown 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 msgr2=0x560b396c4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02cfd0 tx=0x7f7adc005200 comp rx=0 tx=0).stop 2024-03-18T16:08:13.479 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 shutdown_connections 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b396d7b60 0x560b396d9f50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 --2- 172.21.15.150:0/540591430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c4bf0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 >> 172.21.15.150:0/540591430 conn(0x560b396b4a50 msgr2=0x560b396b67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7ad2ffd640 0 client.16086 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 shutdown_connections 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 1 -- 172.21.15.150:0/540591430 wait complete. 2024-03-18T16:08:13.480 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.472+0000 7f7afcbff740 20 client.16086 trim_cache size 0 max 16384 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.486+0000 7f7afcbff740 1 Processor -- start 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.486+0000 7f7afcbff740 1 -- start start 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396dcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41ada30 con 0x560b396b9af0 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41ab300 con 0x560b396c7c30 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7afcbff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41a9150 con 0x560b396ba410 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56564/0 (socket says 172.21.15.150:56564) 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 -- 172.21.15.150:0/1756973549 learned_addr learned my addr 172.21.15.150:0/1756973549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7af37fe640 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396dcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 -- 172.21.15.150:0/1756973549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 msgr2=0x560b396b6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396b6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 -- 172.21.15.150:0/1756973549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 msgr2=0x560b396dcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396dcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 -- 172.21.15.150:0/1756973549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b396c3bc0 con 0x560b396b9af0 2024-03-18T16:08:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02c170 tx=0x7f7adc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79155fd17037ab81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.487+0000 7f7ae97fa640 1 -- 172.21.15.150:0/1756973549 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc004c30 con 0x560b396b9af0 2024-03-18T16:08:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7ae97fa640 1 -- 172.21.15.150:0/1756973549 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc02cb90 con 0x560b396b9af0 2024-03-18T16:08:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7ae97fa640 1 -- 172.21.15.150:0/1756973549 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc062790 con 0x560b396b9af0 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- 172.21.15.150:0/1756973549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 msgr2=0x560b396c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7adc02c170 tx=0x7f7adc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- 172.21.15.150:0/1756973549 shutdown_connections 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396c7c30 0x560b396dcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396ba410 0x560b396b6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 --2- 172.21.15.150:0/1756973549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396b9af0 0x560b396c7320 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- 172.21.15.150:0/1756973549 >> 172.21.15.150:0/1756973549 conn(0x560b396b4a50 msgr2=0x560b396b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- 172.21.15.150:0/1756973549 shutdown_connections 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- 172.21.15.150:0/1756973549 wait complete. 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.488+0000 7f7afcbff740 1 -- start start 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 Processor -- start 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396da750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 0x560b396dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41a97c0 con 0x560b396c7c30 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41aac30 con 0x560b396ba410 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7af41a9e90 con 0x560b396b9af0 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7afcbff740 10 client.0 _ll_register_callbacks cb 0x7f7ae4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56566/0 (socket says 172.21.15.150:56566) 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 0x560b396dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7aeaffd640 1 -- 172.21.15.150:0/3870344051 learned_addr learned my addr 172.21.15.150:0/3870344051 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 -- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 msgr2=0x560b396da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396da750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 -- 172.21.15.150:0/3870344051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 msgr2=0x560b396db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.496 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7af37fe640 1 -- 172.21.15.150:0/3870344051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b3973cda0 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.489+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7af37fe640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 0x560b396dac90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x560b396c33f0 tx=0x7f7ae4020410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b24b98ae3ba918 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4005020 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc02cb90 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f7ae40051c0 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7ae402f050 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 10 client.25856 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560b3973e100 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc02cb90 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f7ae4003680 con 0x560b396ba410 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 10 client.25856 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 10 client.25856 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 20 client.25856 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:13.497 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b3973e830 0x560b39740c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 4 client.25856 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:08:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.490+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b396262d0 con 0x560b3973e830 2024-03-18T16:08:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.491+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b3973e830 0x560b39740c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.491+0000 7f7ad2ffd640 10 client.25856 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.491+0000 7f7aeaffd640 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b3973e830 0x560b39740c50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=943e54903f4cc6e0 server_cookie=36a51236440a245 in_seq=0 out_seq=0 2024-03-18T16:08:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7ad2ffd640 1 -- 172.21.15.150:0/3870344051 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b396262d0 con 0x560b3973e830 2024-03-18T16:08:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7ad2ffd640 10 client.25856 handle_command_reply: tid=0 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 client.25856 shutdown 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 msgr2=0x560b396dac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 0x560b396dac90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x560b396c33f0 tx=0x7f7ae4020410 comp rx=0 tx=0).stop 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 shutdown_connections 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560b3973e830 0x560b39740c50 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560b396c7c30 0x560b396db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7ad2ffd640 0 client.25856 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b396ba410 0x560b396dac90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 --2- 172.21.15.150:0/3870344051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560b396b9af0 0x560b396da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 >> 172.21.15.150:0/3870344051 conn(0x560b396b4a50 msgr2=0x560b396d3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 shutdown_connections 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 1 -- 172.21.15.150:0/3870344051 wait complete. 2024-03-18T16:08:13.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.502+0000 7f7afcbff740 20 client.25856 trim_cache size 0 max 16384 2024-03-18T16:08:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:08:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:08:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:08:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 msgr2=0x7f7ae00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 0x7f7ae00a4ed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7aec005c40 tx=0x7f7aec009f90 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 msgr2=0x7f7af419c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00efd0 tx=0x7f7aec00c6a0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 shutdown_connections 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7ae00a2a10 0x7f7ae00a4ed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af41077a0 0x7f7af419c8e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af4102150 0x7f7af41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 --2- 172.21.15.150:0/2673449074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7af4101790 0x7f7af4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 >> 172.21.15.150:0/2673449074 conn(0x7f7af40fd2a0 msgr2=0x7f7af40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 shutdown_connections 2024-03-18T16:08:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:13.503+0000 7f7af9f3b640 1 -- 172.21.15.150:0/2673449074 wait complete. 2024-03-18T16:08:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:13 smithi191 ceph-mon[36784]: pgmap v253: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-03-18T16:08:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:13 smithi156 ceph-mon[35919]: pgmap v253: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-03-18T16:08:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:13 smithi150 ceph-mon[31946]: pgmap v253: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-03-18T16:08:14.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 827195 10.66 MB/sec execute 299 sec latency 27.281 ms 2024-03-18T16:08:15.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 828652 10.65 MB/sec execute 300 sec latency 29.064 ms 2024-03-18T16:08:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:15 smithi191 ceph-mon[36784]: pgmap v254: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-18T16:08:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:15 smithi156 ceph-mon[35919]: pgmap v254: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-18T16:08:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:15 smithi150 ceph-mon[31946]: pgmap v254: 97 pgs: 97 active+clean; 495 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-18T16:08:16.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 830604 10.65 MB/sec execute 301 sec latency 51.009 ms 2024-03-18T16:08:17.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 832200 10.64 MB/sec execute 302 sec latency 22.454 ms 2024-03-18T16:08:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:17 smithi191 ceph-mon[36784]: pgmap v255: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:17 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:08:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:17 smithi156 ceph-mon[35919]: pgmap v255: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:17 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:08:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:17 smithi150 ceph-mon[31946]: pgmap v255: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:17 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:08:18.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 834284 10.65 MB/sec execute 303 sec latency 19.732 ms 2024-03-18T16:08:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:18 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:18 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:18 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:08:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:18 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:18 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:18 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:08:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:18 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:18 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:18 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:08:19.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 835774 10.64 MB/sec execute 304 sec latency 21.413 ms 2024-03-18T16:08:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:19 smithi191 ceph-mon[36784]: pgmap v256: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:19 smithi156 ceph-mon[35919]: pgmap v256: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:19 smithi150 ceph-mon[31946]: pgmap v256: 97 pgs: 97 active+clean; 507 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-18T16:08:20.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 838041 10.64 MB/sec execute 305 sec latency 31.421 ms 2024-03-18T16:08:21.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 839342 10.63 MB/sec execute 306 sec latency 89.374 ms 2024-03-18T16:08:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:21 smithi191 ceph-mon[36784]: pgmap v257: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 298 op/s 2024-03-18T16:08:22.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:21 smithi156 ceph-mon[35919]: pgmap v257: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 298 op/s 2024-03-18T16:08:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:21 smithi150 ceph-mon[31946]: pgmap v257: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 298 op/s 2024-03-18T16:08:22.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 841528 10.64 MB/sec execute 307 sec latency 90.401 ms 2024-03-18T16:08:23.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 842904 10.62 MB/sec execute 308 sec latency 26.019 ms 2024-03-18T16:08:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:23 smithi191 ceph-mon[36784]: pgmap v258: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:23 smithi156 ceph-mon[35919]: pgmap v258: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:23 smithi150 ceph-mon[31946]: pgmap v258: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:24.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 845229 10.63 MB/sec execute 309 sec latency 25.356 ms 2024-03-18T16:08:25.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 846550 10.62 MB/sec execute 310 sec latency 30.484 ms 2024-03-18T16:08:25.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:25 smithi191 ceph-mon[36784]: pgmap v259: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:25 smithi156 ceph-mon[35919]: pgmap v259: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:25 smithi150 ceph-mon[31946]: pgmap v259: 97 pgs: 97 active+clean; 516 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 190 op/s 2024-03-18T16:08:26.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 848803 10.63 MB/sec execute 311 sec latency 18.048 ms 2024-03-18T16:08:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 -- 172.21.15.150:0/256575885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 msgr2=0x7fd33c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 --2- 172.21.15.150:0/256575885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c105a60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009900 tx=0x7fd32c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 -- 172.21.15.150:0/256575885 shutdown_connections 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 --2- 172.21.15.150:0/256575885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c1012f0 0x7fd33c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 --2- 172.21.15.150:0/256575885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c100930 0x7fd33c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 --2- 172.21.15.150:0/256575885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c105a60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 -- 172.21.15.150:0/256575885 >> 172.21.15.150:0/256575885 conn(0x7fd33c077530 msgr2=0x7fd33c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 -- 172.21.15.150:0/256575885 shutdown_connections 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.595+0000 7fd340a59640 1 -- 172.21.15.150:0/256575885 wait complete. 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 Processor -- start 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 -- start start 2024-03-18T16:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c1012f0 0x7fd33c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c110ba0 con 0x7fd33c105680 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c110a20 con 0x7fd33c100930 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.596+0000 7fd340a59640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c110d20 con 0x7fd33c1012f0 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd333fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c1012f0 0x7fd33c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43908/0 (socket says 172.21.15.150:43908) 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:36026/0 (socket says 172.21.15.150:36026) 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 -- 172.21.15.150:0/3595803075 learned_addr learned my addr 172.21.15.150:0/3595803075 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 -- 172.21.15.150:0/3595803075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c1012f0 msgr2=0x7fd33c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c1012f0 0x7fd33c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 -- 172.21.15.150:0/3595803075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 msgr2=0x7fd33c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 -- 172.21.15.150:0/3595803075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c19d070 con 0x7fd33c105680 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd33ad9d640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd33400ea60 tx=0x7fd33400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd3337fe640 1 -- 172.21.15.150:0/3595803075 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33400ce40 con 0x7fd33c105680 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd3337fe640 1 -- 172.21.15.150:0/3595803075 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd334005020 con 0x7fd33c105680 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33c19d360 con 0x7fd33c105680 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd3337fe640 1 -- 172.21.15.150:0/3595803075 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334010690 con 0x7fd33c105680 2024-03-18T16:08:26.602 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.597+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33c1a4bf0 con 0x7fd33c105680 2024-03-18T16:08:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd3337fe640 1 -- 172.21.15.150:0/3595803075 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fd334005520 con 0x7fd33c105680 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 Processor -- start 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 -- start start 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557130 0x55fd23557510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd235602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fd234bba40 con 0x55fd23560820 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd23464920 con 0x55fd23557130 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.600+0000 7fd343644740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fd234b7ae0 con 0x55fd23557a50 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd3337fe640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 0x7fd3280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd235602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43918/0 (socket says 172.21.15.150:43918) 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 -- 172.21.15.150:0/693956377 learned_addr learned my addr 172.21.15.150:0/693956377 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd33ad9d640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 0x7fd3280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557130 0x55fd23557510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd3337fe640 1 -- 172.21.15.150:0/3595803075 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fd3340cf330 con 0x7fd33c105680 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 -- 172.21.15.150:0/693956377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 msgr2=0x55fd235602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd235602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 -- 172.21.15.150:0/693956377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557130 msgr2=0x55fd23557510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557130 0x55fd23557510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 -- 172.21.15.150:0/693956377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2347c460 con 0x55fd23560820 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd33a59c640 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd235602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd333fff640 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd324011470 tx=0x7fd324009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0fd3078191293c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd33ad9d640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 0x7fd3280a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd33409b690 tx=0x7fd334097590 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd330ff9640 1 -- 172.21.15.150:0/693956377 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324004050 con 0x55fd23560820 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd330ff9640 1 -- 172.21.15.150:0/693956377 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3240041f0 con 0x55fd23560820 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.601+0000 7fd330ff9640 1 -- 172.21.15.150:0/693956377 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32401d600 con 0x55fd23560820 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- 172.21.15.150:0/693956377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 msgr2=0x55fd23562c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd324011470 tx=0x7fd324009880 comp rx=0 tx=0).stop 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- 172.21.15.150:0/693956377 shutdown_connections 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23562c10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd235602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 --2- 172.21.15.150:0/693956377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557130 0x55fd23557510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- 172.21.15.150:0/693956377 >> 172.21.15.150:0/693956377 conn(0x55fd23551a50 msgr2=0x55fd23551e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- 172.21.15.150:0/693956377 shutdown_connections 2024-03-18T16:08:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- 172.21.15.150:0/693956377 wait complete. 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.602+0000 7fd343644740 1 -- start start 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 Processor -- start 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd2355de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd2355e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fd23563450 con 0x55fd23557130 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fd23477d70 con 0x55fd23560820 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fd235635d0 con 0x55fd23557a50 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 10 client.0 _ll_register_callbacks cb 0x55fd231cfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43920/0 (socket says 172.21.15.150:43920) 2024-03-18T16:08:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 -- 172.21.15.150:0/68579999 learned_addr learned my addr 172.21.15.150:0/68579999 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd33a59c640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd2355de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd2355e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 -- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 msgr2=0x55fd2355de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd2355de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 -- 172.21.15.150:0/68579999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 msgr2=0x55fd2355e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd2355e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 -- 172.21.15.150:0/68579999 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd231d0210 con 0x55fd23557130 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd333fff640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd324006010 tx=0x7fd324005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e560ca2ad81f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd321ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324022030 con 0x55fd23557130 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fd324005250 con 0x55fd23557130 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.603+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3240041f0 con 0x55fd23557130 2024-03-18T16:08:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd324031050 con 0x55fd23557130 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 10 client.16104 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fd2329c620 con 0x55fd23557130 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3240041f0 con 0x55fd23557130 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd32401e410 con 0x55fd23557130 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 10 client.16104 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 10 client.16104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 20 client.16104 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd23570970 0x55fd23572d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 4 client.16104 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd234c8f80 con 0x55fd23570970 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.604+0000 7fd33a59c640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd23570970 0x55fd23572d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.605+0000 7fd321ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:26.615 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.605+0000 7fd33a59c640 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd23570970 0x55fd23572d60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9fcf54794c962 server_cookie=6697c48bd173e44d in_seq=0 out_seq=0 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.605+0000 7fd321ffb640 1 -- 172.21.15.150:0/68579999 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd234c8f80 con 0x55fd23570970 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.605+0000 7fd321ffb640 10 client.16104 handle_command_reply: tid=0 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.605+0000 7fd343644740 1 client.16104 shutdown 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 msgr2=0x55fd2355d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd324006010 tx=0x7fd324005850 comp rx=0 tx=0).stop 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 shutdown_connections 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd23570970 0x55fd23572d60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd2355e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557a50 0x55fd2355de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 --2- 172.21.15.150:0/68579999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557130 0x55fd2355d920 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd321ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 >> 172.21.15.150:0/68579999 conn(0x55fd23551a50 msgr2=0x55fd235537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 shutdown_connections 2024-03-18T16:08:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 1 -- 172.21.15.150:0/68579999 wait complete. 2024-03-18T16:08:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.606+0000 7fd343644740 20 client.16104 trim_cache size 0 max 16384 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 Processor -- start 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 -- start start 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd2356c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd23552e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1afce0 con 0x55fd23557a50 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1ad5b0 con 0x55fd23560820 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.620+0000 7fd343644740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1ab420 con 0x55fd23557130 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43928/0 (socket says 172.21.15.150:43928) 2024-03-18T16:08:26.627 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 -- 172.21.15.150:0/3617386463 learned_addr learned my addr 172.21.15.150:0/3617386463 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd333fff640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd2356c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd333fff640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd23552e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 -- 172.21.15.150:0/3617386463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 msgr2=0x55fd2356c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd2356c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 -- 172.21.15.150:0/3617386463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 msgr2=0x55fd23552e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd23552e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 -- 172.21.15.150:0/3617386463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2354bfc0 con 0x55fd23557a50 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd333fff640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd23552e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd32c035000 tx=0x7fd32c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5815629b67a57466 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd323fff640 1 -- 172.21.15.150:0/3617386463 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c004040 con 0x55fd23557a50 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd323fff640 1 -- 172.21.15.150:0/3617386463 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd32c0041e0 con 0x55fd23557a50 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.621+0000 7fd323fff640 1 -- 172.21.15.150:0/3617386463 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c005520 con 0x55fd23557a50 2024-03-18T16:08:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- 172.21.15.150:0/3617386463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 msgr2=0x55fd2355d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd32c035000 tx=0x7fd32c005fd0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- 172.21.15.150:0/3617386463 shutdown_connections 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23560820 0x55fd23552e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23557a50 0x55fd2355d200 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- 172.21.15.150:0/3617386463 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd2356c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- 172.21.15.150:0/3617386463 >> 172.21.15.150:0/3617386463 conn(0x55fd23551a50 msgr2=0x55fd23550bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- 172.21.15.150:0/3617386463 shutdown_connections 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- 172.21.15.150:0/3617386463 wait complete. 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- start start 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 Processor -- start 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd23568a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 0x55fd23563de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23564320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1aba70 con 0x55fd23560820 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1acee0 con 0x55fd23557a50 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c1ac140 con 0x55fd23557130 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd343644740 10 client.0 _ll_register_callbacks cb 0x7fd324005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23564320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd333fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd23568a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd333fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd23568a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:47800/0 (socket says 172.21.15.150:47800) 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd333fff640 1 -- 172.21.15.150:0/3414132625 learned_addr learned my addr 172.21.15.150:0/3414132625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.622+0000 7fd33a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 0x55fd23563de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 -- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 msgr2=0x55fd23568a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd23568a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 -- 172.21.15.150:0/3414132625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 msgr2=0x55fd23563de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 0x55fd23563de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 -- 172.21.15.150:0/3414132625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd23564b20 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 0x55fd23563de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd333fff640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23564320 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd32401e7e0 tx=0x7fd324005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8178b327b0935044 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd321ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324022040 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fd324011220 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd32c0041e0 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd324005030 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd343644740 10 client.16110 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fd235d69d0 con 0x55fd23560820 2024-03-18T16:08:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.623+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd32c0041e0 con 0x55fd23560820 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd324004ba0 con 0x55fd23560820 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 10 client.16110 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 20 client.16110 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd235d7080 0x55fd235d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 4 client.16110 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd234c32d0 con 0x55fd235d7080 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd235d7080 0x55fd235d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd321ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.624+0000 7fd33a59c640 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd235d7080 0x55fd235d94a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a98f2613478ea5 server_cookie=2469cbb529d329f8 in_seq=0 out_seq=0 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.634+0000 7fd321ffb640 1 -- 172.21.15.150:0/3414132625 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd234c32d0 con 0x55fd235d7080 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.634+0000 7fd321ffb640 10 client.16110 handle_command_reply: tid=0 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.634+0000 7fd343644740 1 client.16110 shutdown 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 msgr2=0x55fd23564320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23564320 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd32401e7e0 tx=0x7fd324005850 comp rx=0 tx=0).stop 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 shutdown_connections 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fd235d7080 0x55fd235d94a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fd23560820 0x55fd23564320 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fd23557a50 0x55fd23563de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 --2- 172.21.15.150:0/3414132625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fd23557130 0x55fd23568a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd321ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 >> 172.21.15.150:0/3414132625 conn(0x55fd23551a50 msgr2=0x55fd23550320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 shutdown_connections 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 1 -- 172.21.15.150:0/3414132625 wait complete. 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd343644740 20 client.16110 trim_cache size 0 max 16384 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:08:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:08:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 msgr2=0x7fd3280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 0x7fd3280a4ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd33409b690 tx=0x7fd334097590 comp rx=0 tx=0).stop 2024-03-18T16:08:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 msgr2=0x7fd33c19c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd33400ea60 tx=0x7fd33400ef20 comp rx=0 tx=0).stop 2024-03-18T16:08:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 shutdown_connections 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd3280a2a10 0x7fd3280a4ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c105680 0x7fd33c19c940 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd33c1012f0 0x7fd33c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 --2- 172.21.15.150:0/3595803075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33c100930 0x7fd33c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 >> 172.21.15.150:0/3595803075 conn(0x7fd33c077530 msgr2=0x7fd33c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 shutdown_connections 2024-03-18T16:08:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:26.635+0000 7fd340a59640 1 -- 172.21.15.150:0/3595803075 wait complete. 2024-03-18T16:08:27.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 849964 10.61 MB/sec execute 312 sec latency 92.469 ms 2024-03-18T16:08:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:27 smithi156 ceph-mon[35919]: pgmap v260: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 281 op/s 2024-03-18T16:08:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:27 smithi150 ceph-mon[31946]: pgmap v260: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 281 op/s 2024-03-18T16:08:28.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:27 smithi191 ceph-mon[36784]: pgmap v260: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 281 op/s 2024-03-18T16:08:28.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 851482 10.60 MB/sec execute 313 sec latency 19.408 ms 2024-03-18T16:08:29.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 853055 10.59 MB/sec execute 314 sec latency 26.952 ms 2024-03-18T16:08:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:29 smithi156 ceph-mon[35919]: pgmap v261: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:29 smithi150 ceph-mon[31946]: pgmap v261: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:30.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:29 smithi191 ceph-mon[36784]: pgmap v261: 97 pgs: 97 active+clean; 525 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:30.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 855011 10.59 MB/sec execute 315 sec latency 33.981 ms 2024-03-18T16:08:31.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 856633 10.58 MB/sec execute 316 sec latency 16.863 ms 2024-03-18T16:08:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:31 smithi156 ceph-mon[35919]: pgmap v262: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-03-18T16:08:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:31 smithi150 ceph-mon[31946]: pgmap v262: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-03-18T16:08:32.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:31 smithi191 ceph-mon[36784]: pgmap v262: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 277 op/s 2024-03-18T16:08:32.334 INFO:tasks.workunit.client.0.smithi150.stdout: 1 858208 10.57 MB/sec execute 317 sec latency 123.419 ms 2024-03-18T16:08:33.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 860076 10.58 MB/sec execute 318 sec latency 18.248 ms 2024-03-18T16:08:34.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:33 smithi156 ceph-mon[35919]: pgmap v263: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:33 smithi150 ceph-mon[31946]: pgmap v263: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:34.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:33 smithi191 ceph-mon[36784]: pgmap v263: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:34.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 861012 10.56 MB/sec execute 319 sec latency 28.218 ms 2024-03-18T16:08:35.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:34 smithi150 ceph-mon[31946]: pgmap v264: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:35.114 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:34 smithi191 ceph-mon[36784]: pgmap v264: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:35.269 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:34 smithi156 ceph-mon[35919]: pgmap v264: 97 pgs: 97 active+clean; 497 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 184 op/s 2024-03-18T16:08:35.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 862948 10.57 MB/sec execute 320 sec latency 16.186 ms 2024-03-18T16:08:36.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 864303 10.55 MB/sec execute 321 sec latency 28.853 ms 2024-03-18T16:08:37.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 866071 10.55 MB/sec execute 322 sec latency 164.662 ms 2024-03-18T16:08:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:37 smithi191 ceph-mon[36784]: pgmap v265: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-18T16:08:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:37 smithi156 ceph-mon[35919]: pgmap v265: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-18T16:08:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:37 smithi150 ceph-mon[31946]: pgmap v265: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-18T16:08:38.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 867514 10.53 MB/sec execute 323 sec latency 22.284 ms 2024-03-18T16:08:39.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 869679 10.54 MB/sec execute 324 sec latency 19.323 ms 2024-03-18T16:08:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 -- 172.21.15.150:0/3518098298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 msgr2=0x7fecd810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 --2- 172.21.15.150:0/3518098298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd810d510 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009ca0 tx=0x7fecd002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 -- 172.21.15.150:0/3518098298 shutdown_connections 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 --2- 172.21.15.150:0/3518098298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd810d510 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 --2- 172.21.15.150:0/3518098298 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 0x7fecd8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 --2- 172.21.15.150:0/3518098298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 0x7fecd8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 -- 172.21.15.150:0/3518098298 >> 172.21.15.150:0/3518098298 conn(0x7fecd8077530 msgr2=0x7fecd8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 -- 172.21.15.150:0/3518098298 shutdown_connections 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 -- 172.21.15.150:0/3518098298 wait complete. 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.724+0000 7fece0722640 1 Processor -- start 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- start start 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 0x7fecd8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 0x7fecd819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fecd8110ba0 con 0x7fecd81012f0 2024-03-18T16:08:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecd8110a20 con 0x7fecd8105680 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fecd8110d20 con 0x7fecd8100930 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecde4be640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 0x7fecd8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56566/0 (socket says 172.21.15.150:56566) 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 -- 172.21.15.150:0/1995539396 learned_addr learned my addr 172.21.15.150:0/1995539396 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecdecbf640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 0x7fecd819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 -- 172.21.15.150:0/1995539396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 msgr2=0x7fecd8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 0x7fecd8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 -- 172.21.15.150:0/1995539396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 msgr2=0x7fecd819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 0x7fecd819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 -- 172.21.15.150:0/1995539396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd819d030 con 0x7fecd81012f0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fecddcbd640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fecc800e9f0 tx=0x7fecc800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7feccf7fe640 1 -- 172.21.15.150:0/1995539396 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc800cd80 con 0x7fecd81012f0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7feccf7fe640 1 -- 172.21.15.150:0/1995539396 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc8005500 con 0x7fecd81012f0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7feccf7fe640 1 -- 172.21.15.150:0/1995539396 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8010690 con 0x7fecd81012f0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd819d320 con 0x7fecd81012f0 2024-03-18T16:08:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.725+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd81a4bb0 con 0x7fecd81012f0 2024-03-18T16:08:39.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7feccf7fe640 1 -- 172.21.15.150:0/1995539396 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fecc80046c0 con 0x7fecd81012f0 2024-03-18T16:08:39.735 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 Processor -- start 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 -- start start 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd3361130 0x55efd3361510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 0x55efd336cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55efd32c5a40 con 0x55efd3361a50 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55efd326e920 con 0x55efd336a820 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7fece33eb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd32c1ae0 con 0x55efd3361130 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.728+0000 7feccf7fe640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 0x7fecc40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecdecbf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd3361130 0x55efd3361510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56574/0 (socket says 172.21.15.150:56574) 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecdecbf640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 0x7fecc40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 -- 172.21.15.150:0/3530471994 learned_addr learned my addr 172.21.15.150:0/3530471994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:39.736 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 0x55efd336cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7feccf7fe640 1 -- 172.21.15.150:0/1995539396 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fecc80cf250 con 0x7fecd81012f0 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 -- 172.21.15.150:0/3530471994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd3361130 msgr2=0x55efd3361510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd3361130 0x55efd3361510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 -- 172.21.15.150:0/3530471994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 msgr2=0x55efd336cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 0x55efd336cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 -- 172.21.15.150:0/3530471994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd3286460 con 0x55efd3361a50 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecddcbd640 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 0x55efd336cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecdecbf640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 0x7fecc40a4ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fecd0004af0 tx=0x7fecd0009380 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecde4be640 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009cf0 tx=0x7fecd4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f28e2e38dedba87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecccff9640 1 -- 172.21.15.150:0/3530471994 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd4004220 con 0x55efd3361a50 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecccff9640 1 -- 172.21.15.150:0/3530471994 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecd40043c0 con 0x55efd3361a50 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.729+0000 7fecccff9640 1 -- 172.21.15.150:0/3530471994 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd40188a0 con 0x55efd3361a50 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- 172.21.15.150:0/3530471994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 msgr2=0x55efd336a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009cf0 tx=0x7fecd4011810 comp rx=0 tx=0).stop 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- 172.21.15.150:0/3530471994 shutdown_connections 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd336a820 0x55efd336cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336a2e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 --2- 172.21.15.150:0/3530471994 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd3361130 0x55efd3361510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- 172.21.15.150:0/3530471994 >> 172.21.15.150:0/3530471994 conn(0x55efd335ba50 msgr2=0x55efd335be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- 172.21.15.150:0/3530471994 shutdown_connections 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- 172.21.15.150:0/3530471994 wait complete. 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 -- start start 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.730+0000 7fece33eb740 1 Processor -- start 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd33677b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55efd336d450 con 0x55efd3361a50 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55efd3281d70 con 0x55efd3361130 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55efd336d5d0 con 0x55efd336a820 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 10 client.0 _ll_register_callbacks cb 0x55efd2fd9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56582/0 (socket says 172.21.15.150:56582) 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecde4be640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd33677b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 -- 172.21.15.150:0/3741506153 learned_addr learned my addr 172.21.15.150:0/3741506153 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecdecbf640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 -- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 msgr2=0x55efd3368230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 -- 172.21.15.150:0/3741506153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 msgr2=0x55efd33677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd33677b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 -- 172.21.15.150:0/3741506153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd2fda210 con 0x55efd3361a50 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecde4be640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd33677b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecddcbd640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fecc809b5b0 tx=0x7fecc809b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500cd8e0b6b2eb1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc809bfb0 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fecc8086110 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.731+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecd40043c0 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fecc809b8b0 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 10 client.16122 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55efd30a6620 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecd40043c0 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fecc80a3bc0 con 0x55efd3361a50 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 10 client.16122 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 20 client.16122 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd337a960 0x55efd337cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 4 client.16122 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:08:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efd32d2f80 con 0x55efd337a960 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecde4be640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd337a960 0x55efd337cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecc1ffb640 10 client.16122 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.732+0000 7fecde4be640 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd337a960 0x55efd337cd90 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae40534b49773cf server_cookie=ed298e7451ec9095 in_seq=0 out_seq=0 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fecc1ffb640 1 -- 172.21.15.150:0/3741506153 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55efd32d2f80 con 0x55efd337a960 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fecc1ffb640 10 client.16122 handle_command_reply: tid=0 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 client.16122 shutdown 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 msgr2=0x55efd3367cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fecc809b5b0 tx=0x7fecc809b740 comp rx=0 tx=0).stop 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 shutdown_connections 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd337a960 0x55efd337cd90 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fecc1ffb640 0 client.16122 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd3367cf0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 --2- 172.21.15.150:0/3741506153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd33677b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 >> 172.21.15.150:0/3741506153 conn(0x55efd335ba50 msgr2=0x55efd335d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 shutdown_connections 2024-03-18T16:08:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 1 -- 172.21.15.150:0/3741506153 wait complete. 2024-03-18T16:08:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.733+0000 7fece33eb740 20 client.16122 trim_cache size 0 max 16384 2024-03-18T16:08:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 Processor -- start 2024-03-18T16:08:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 -- start start 2024-03-18T16:08:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd3369ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.754 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81afc50 con 0x55efd3361a50 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81ad520 con 0x55efd3361130 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fece33eb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81ab390 con 0x55efd336a820 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56596/0 (socket says 172.21.15.150:56596) 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecdecbf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 -- 172.21.15.150:0/402485063 learned_addr learned my addr 172.21.15.150:0/402485063 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecde4be640 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd3369ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 -- 172.21.15.150:0/402485063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 msgr2=0x55efd3368000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 -- 172.21.15.150:0/402485063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 msgr2=0x55efd3369ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd3369ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 -- 172.21.15.150:0/402485063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efd3366760 con 0x55efd3361a50 2024-03-18T16:08:39.755 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecddcbd640 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fecc8098280 tx=0x7fecc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7906acbd668c0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecc3fff640 1 -- 172.21.15.150:0/402485063 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc80ac070 con 0x55efd3361a50 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.749+0000 7fecc3fff640 1 -- 172.21.15.150:0/402485063 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc809bfb0 con 0x55efd3361a50 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fecc3fff640 1 -- 172.21.15.150:0/402485063 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc80a3bc0 con 0x55efd3361a50 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- 172.21.15.150:0/402485063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 msgr2=0x55efd33761f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fecc8098280 tx=0x7fecc8004a40 comp rx=0 tx=0).stop 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- 172.21.15.150:0/402485063 shutdown_connections 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd3368000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd33761f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 --2- 172.21.15.150:0/402485063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd3369ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- 172.21.15.150:0/402485063 >> 172.21.15.150:0/402485063 conn(0x55efd335ba50 msgr2=0x55efd335be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- 172.21.15.150:0/402485063 shutdown_connections 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- 172.21.15.150:0/402485063 wait complete. 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 -- start start 2024-03-18T16:08:39.756 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.750+0000 7fece33eb740 1 Processor -- start 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd336e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd336ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81ab9e0 con 0x55efd3361a50 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81ace50 con 0x55efd3361130 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fecd81ac0b0 con 0x55efd336a820 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 10 client.0 _ll_register_callbacks cb 0x7fecc800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56610/0 (socket says 172.21.15.150:56610) 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 -- 172.21.15.150:0/2471216871 learned_addr learned my addr 172.21.15.150:0/2471216871 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecdecbf640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd336ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecde4be640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd336e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 -- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 msgr2=0x55efd336ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.757 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd336ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 -- 172.21.15.150:0/2471216871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 msgr2=0x55efd336e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd336e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 -- 172.21.15.150:0/2471216871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efd3361880 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecddcbd640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fecc80a3b20 tx=0x7fecc8009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33636b353a9a15fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc80ac070 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7fecc809cbe0 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecc809bfb0 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fecc8097b50 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 10 client.16128 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.751+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55efd33e1030 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc809bfb0 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fecc80b43d0 con 0x55efd3361a50 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 10 client.16128 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 20 client.16128 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd33e13f0 0x55efd33e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 4 client.16128 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55efd32cd2d0 con 0x55efd33e13f0 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fecde4be640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd33e13f0 0x55efd33e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fecc1ffb640 10 client.16128 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:39.759 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.752+0000 7fecde4be640 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd33e13f0 0x55efd33e3810 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acccbb55370d6402 server_cookie=5d39f24f99d7fa in_seq=0 out_seq=0 2024-03-18T16:08:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fecc1ffb640 1 -- 172.21.15.150:0/2471216871 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55efd32cd2d0 con 0x55efd33e13f0 2024-03-18T16:08:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fecc1ffb640 10 client.16128 handle_command_reply: tid=0 2024-03-18T16:08:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 client.16128 shutdown 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 msgr2=0x55efd336e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fecc80a3b20 tx=0x7fecc8009720 comp rx=0 tx=0).stop 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 shutdown_connections 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55efd33e13f0 0x55efd33e3810 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55efd336a820 0x55efd336ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efd3361a50 0x55efd336e8c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 --2- 172.21.15.150:0/2471216871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55efd3361130 0x55efd336e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fecc1ffb640 0 client.16128 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 >> 172.21.15.150:0/2471216871 conn(0x55efd335ba50 msgr2=0x55efd335abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 shutdown_connections 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.763+0000 7fece33eb740 1 -- 172.21.15.150:0/2471216871 wait complete. 2024-03-18T16:08:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece33eb740 20 client.16128 trim_cache size 0 max 16384 2024-03-18T16:08:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:08:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:08:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:08:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 msgr2=0x7fecc40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 0x7fecc40a4ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fecd0004af0 tx=0x7fecd0009380 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 msgr2=0x7fecd81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fecc800e9f0 tx=0x7fecc800edf0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 shutdown_connections 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fecc40a2a10 0x7fecc40a4ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecd8105680 0x7fecd819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd81012f0 0x7fecd81985d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 --2- 172.21.15.150:0/1995539396 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fecd8100930 0x7fecd8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 >> 172.21.15.150:0/1995539396 conn(0x7fecd8077530 msgr2=0x7fecd8068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:39.768 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 shutdown_connections 2024-03-18T16:08:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:39.764+0000 7fece0722640 1 -- 172.21.15.150:0/1995539396 wait complete. 2024-03-18T16:08:39.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:39 smithi191 ceph-mon[36784]: pgmap v266: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 172 op/s 2024-03-18T16:08:40.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:39 smithi156 ceph-mon[35919]: pgmap v266: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 172 op/s 2024-03-18T16:08:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:39 smithi150 ceph-mon[31946]: pgmap v266: 97 pgs: 97 active+clean; 504 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 172 op/s 2024-03-18T16:08:40.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 871288 10.54 MB/sec execute 325 sec latency 18.818 ms 2024-03-18T16:08:41.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 873585 10.55 MB/sec execute 326 sec latency 18.258 ms 2024-03-18T16:08:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:41 smithi191 ceph-mon[36784]: pgmap v267: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 262 op/s 2024-03-18T16:08:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:41 smithi156 ceph-mon[35919]: pgmap v267: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 262 op/s 2024-03-18T16:08:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:41 smithi150 ceph-mon[31946]: pgmap v267: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 262 op/s 2024-03-18T16:08:42.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 874840 10.53 MB/sec execute 327 sec latency 76.728 ms 2024-03-18T16:08:43.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 876926 10.54 MB/sec execute 328 sec latency 40.872 ms 2024-03-18T16:08:43.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:43 smithi191 ceph-mon[36784]: pgmap v268: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:43 smithi156 ceph-mon[35919]: pgmap v268: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:43 smithi150 ceph-mon[31946]: pgmap v268: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:44.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 878380 10.53 MB/sec execute 329 sec latency 20.785 ms 2024-03-18T16:08:45.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 880550 10.53 MB/sec execute 330 sec latency 25.399 ms 2024-03-18T16:08:45.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:45 smithi191 ceph-mon[36784]: pgmap v269: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:45 smithi156 ceph-mon[35919]: pgmap v269: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:45 smithi150 ceph-mon[31946]: pgmap v269: 97 pgs: 97 active+clean; 513 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-18T16:08:46.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 882006 10.52 MB/sec execute 331 sec latency 20.038 ms 2024-03-18T16:08:47.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 883683 10.51 MB/sec execute 332 sec latency 70.181 ms 2024-03-18T16:08:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:47 smithi191 ceph-mon[36784]: pgmap v270: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-18T16:08:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:47 smithi156 ceph-mon[35919]: pgmap v270: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-18T16:08:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:47 smithi150 ceph-mon[31946]: pgmap v270: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-18T16:08:48.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 885086 10.50 MB/sec execute 333 sec latency 20.944 ms 2024-03-18T16:08:49.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 886948 10.50 MB/sec execute 334 sec latency 28.845 ms 2024-03-18T16:08:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:49 smithi191 ceph-mon[36784]: pgmap v271: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:08:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:49 smithi156 ceph-mon[35919]: pgmap v271: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:08:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:49 smithi150 ceph-mon[31946]: pgmap v271: 97 pgs: 97 active+clean; 522 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 181 op/s 2024-03-18T16:08:50.335 INFO:tasks.workunit.client.0.smithi150.stdout: 1 888660 10.50 MB/sec execute 335 sec latency 17.175 ms 2024-03-18T16:08:51.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 890739 10.50 MB/sec execute 336 sec latency 26.348 ms 2024-03-18T16:08:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:51 smithi191 ceph-mon[36784]: pgmap v272: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 268 op/s 2024-03-18T16:08:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:51 smithi156 ceph-mon[35919]: pgmap v272: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 268 op/s 2024-03-18T16:08:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:51 smithi150 ceph-mon[31946]: pgmap v272: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 268 op/s 2024-03-18T16:08:52.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 892023 10.49 MB/sec execute 337 sec latency 62.918 ms 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 -- 172.21.15.150:0/2356601067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c105660 msgr2=0x7f199c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2356601067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c105660 0x7f199c105a40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1998009900 tx=0x7f199802bc30 comp rx=0 tx=0).stop 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 -- 172.21.15.150:0/2356601067 shutdown_connections 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2356601067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 0x7f199c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2356601067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c100930 0x7f199c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2356601067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c105660 0x7f199c105a40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 -- 172.21.15.150:0/2356601067 >> 172.21.15.150:0/2356601067 conn(0x7f199c0774b0 msgr2=0x7f199c0778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 -- 172.21.15.150:0/2356601067 shutdown_connections 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.868+0000 7f19a3d10640 1 -- 172.21.15.150:0/2356601067 wait complete. 2024-03-18T16:08:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 Processor -- start 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 -- start start 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 0x7f199c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c105660 0x7f199c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f199c110b70 con 0x7f199c100930 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1109f0 con 0x7f199c105660 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a3d10640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f199c110cf0 con 0x7f199c1012d0 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50780/0 (socket says 172.21.15.150:50780) 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a1aac640 1 -- 172.21.15.150:0/2381642862 learned_addr learned my addr 172.21.15.150:0/2381642862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a22ad640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c105660 0x7f199c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a12ab640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 0x7f199c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.869+0000 7f19a1aac640 1 -- 172.21.15.150:0/2381642862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 msgr2=0x7f199c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 0x7f199c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a1aac640 1 -- 172.21.15.150:0/2381642862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c105660 msgr2=0x7f199c19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c105660 0x7f199c19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a1aac640 1 -- 172.21.15.150:0/2381642862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c19d010 con 0x7f199c100930 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1998006010 tx=0x7f199802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f1992ffd640 1 -- 172.21.15.150:0/2381642862 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998041070 con 0x7f199c100930 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f1992ffd640 1 -- 172.21.15.150:0/2381642862 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1998004450 con 0x7f199c100930 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c19d2a0 con 0x7f199c100930 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f1992ffd640 1 -- 172.21.15.150:0/2381642862 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998003e00 con 0x7f199c100930 2024-03-18T16:08:52.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.870+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c1a4b90 con 0x7f199c100930 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.872+0000 7f1992ffd640 1 -- 172.21.15.150:0/2381642862 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f1998044070 con 0x7f199c100930 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 Processor -- start 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 -- start start 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f1992ffd640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 0x7f19880a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d9229822e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922982820 0x55d922984c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d9228dda40 con 0x55d922979130 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d922886920 con 0x55d922982820 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a69da740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d9228d9ae0 con 0x55d922979a50 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.873+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a22ad640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 0x7f19880a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50786/0 (socket says 172.21.15.150:50786) 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a1aac640 1 -- 172.21.15.150:0/3797905082 learned_addr learned my addr 172.21.15.150:0/3797905082 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d9229822e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a22ad640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922982820 0x55d922984c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f1992ffd640 1 -- 172.21.15.150:0/2381642862 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f19980f0720 con 0x7f199c100930 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 -- 172.21.15.150:0/3797905082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922982820 msgr2=0x55d922984c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922982820 0x55d922984c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 -- 172.21.15.150:0/3797905082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 msgr2=0x55d922979510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 -- 172.21.15.150:0/3797905082 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d92289e460 con 0x55d922979a50 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a1aac640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a22ad640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 0x7f19880a4f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1994005fd0 tx=0x7f1994009f10 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f19a12ab640 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d9229822e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f198c009bf0 tx=0x7f198c011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a55dfb04e4bc040 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f1987fff640 1 -- 172.21.15.150:0/3797905082 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c0042f0 con 0x55d922979a50 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f1987fff640 1 -- 172.21.15.150:0/3797905082 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c004490 con 0x55d922979a50 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.874+0000 7f1987fff640 1 -- 172.21.15.150:0/3797905082 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c019af0 con 0x55d922979a50 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- 172.21.15.150:0/3797905082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 msgr2=0x55d9229822e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d9229822e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f198c009bf0 tx=0x7f198c011440 comp rx=0 tx=0).stop 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- 172.21.15.150:0/3797905082 shutdown_connections 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922982820 0x55d922984c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d9229822e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 --2- 172.21.15.150:0/3797905082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979130 0x55d922979510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- 172.21.15.150:0/3797905082 >> 172.21.15.150:0/3797905082 conn(0x55d922973a50 msgr2=0x55d922973e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- 172.21.15.150:0/3797905082 shutdown_connections 2024-03-18T16:08:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- 172.21.15.150:0/3797905082 wait complete. 2024-03-18T16:08:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.875+0000 7f19a69da740 1 -- start start 2024-03-18T16:08:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 Processor -- start 2024-03-18T16:08:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d92297f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922982820 0x55d922980290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d922985450 con 0x55d922979a50 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d922899d70 con 0x55d922979130 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55d9229855d0 con 0x55d922982820 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a69da740 10 client.0 _ll_register_callbacks cb 0x55d9225f1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a12ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d92297f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a12ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50802/0 (socket says 172.21.15.150:50802) 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a12ab640 1 -- 172.21.15.150:0/2278649660 learned_addr learned my addr 172.21.15.150:0/2278649660 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a22ad640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922982820 0x55d922980290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 -- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922982820 msgr2=0x55d922980290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922982820 0x55d922980290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 -- 172.21.15.150:0/2278649660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 msgr2=0x55d92297fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a1aac640 1 -- 172.21.15.150:0/2278649660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9225f2210 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.876+0000 7f19a12ab640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a1aac640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d92297f810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19980ee030 tx=0x7f19980bc960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=651f8ab5df1aa8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199804b040 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f19980c2190 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f198c004490 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f199802cd20 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 10 client.25904 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9226be620 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c004490 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f19980b95b0 con 0x55d922979130 2024-03-18T16:08:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 10 client.25904 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 10 client.25904 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 20 client.25904 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d922992980 0x55d922994df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 4 client.25904 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9228eaf80 con 0x55d922992980 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.877+0000 7f19a12ab640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d922992980 0x55d922994df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.878+0000 7f19a12ab640 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d922992980 0x55d922994df0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a09087e25f529f37 server_cookie=4e94fdc874f04ca9 in_seq=0 out_seq=0 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.878+0000 7f19857fa640 10 client.25904 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.878+0000 7f19857fa640 1 -- 172.21.15.150:0/2278649660 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9228eaf80 con 0x55d922992980 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.878+0000 7f19857fa640 10 client.25904 handle_command_reply: tid=0 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.878+0000 7f19a69da740 1 client.25904 shutdown 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 msgr2=0x55d92297f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d92297f810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19980ee030 tx=0x7f19980bc960 comp rx=0 tx=0).stop 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 shutdown_connections 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d922992980 0x55d922994df0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922982820 0x55d922980290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19857fa640 0 client.25904 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922979a50 0x55d92297fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 --2- 172.21.15.150:0/2278649660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d92297f810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 >> 172.21.15.150:0/2278649660 conn(0x55d922973a50 msgr2=0x55d9229757a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 shutdown_connections 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 1 -- 172.21.15.150:0/2278649660 wait complete. 2024-03-18T16:08:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.879+0000 7f19a69da740 20 client.25904 trim_cache size 0 max 16384 2024-03-18T16:08:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 Processor -- start 2024-03-18T16:08:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 -- start start 2024-03-18T16:08:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d922981f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d92298e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1afba0 con 0x55d922982820 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1ad470 con 0x55d922979130 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a69da740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1ab2e0 con 0x55d922979a50 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a22ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a22ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50804/0 (socket says 172.21.15.150:50804) 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a22ad640 1 -- 172.21.15.150:0/693302645 learned_addr learned my addr 172.21.15.150:0/693302645 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a1aac640 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d922981f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.894+0000 7f19a12ab640 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d92298e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 -- 172.21.15.150:0/693302645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 msgr2=0x55d92298e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d92298e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 -- 172.21.15.150:0/693302645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 msgr2=0x55d922981f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d922981f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 -- 172.21.15.150:0/693302645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d922973880 con 0x55d922982820 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a22ad640 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f199400ffa0 tx=0x7f1994021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e22aaaa535db72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19877fe640 1 -- 172.21.15.150:0/693302645 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994030070 con 0x55d922982820 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19877fe640 1 -- 172.21.15.150:0/693302645 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1994011c50 con 0x55d922982820 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19877fe640 1 -- 172.21.15.150:0/693302645 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940203f0 con 0x55d922982820 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a69da740 1 -- 172.21.15.150:0/693302645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 msgr2=0x55d92299a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a69da740 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f199400ffa0 tx=0x7f1994021000 comp rx=0 tx=0).stop 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a69da740 1 -- 172.21.15.150:0/693302645 shutdown_connections 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.895+0000 7f19a69da740 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d92299a700 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979a50 0x55d92298e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 --2- 172.21.15.150:0/693302645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979130 0x55d922981f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- 172.21.15.150:0/693302645 >> 172.21.15.150:0/693302645 conn(0x55d922973a50 msgr2=0x55d922975730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- 172.21.15.150:0/693302645 shutdown_connections 2024-03-18T16:08:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- 172.21.15.150:0/693302645 wait complete. 2024-03-18T16:08:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- start start 2024-03-18T16:08:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 Processor -- start 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979130 0x55d9229862a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 0x55d9229867e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1ab930 con 0x55d922982820 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1acda0 con 0x55d922979a50 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f199c1ac000 con 0x55d922979130 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a69da740 10 client.0 _ll_register_callbacks cb 0x7f19980edf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:08:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a22ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a22ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50818/0 (socket says 172.21.15.150:50818) 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a1aac640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979130 0x55d9229862a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a22ad640 1 -- 172.21.15.150:0/1976247200 learned_addr learned my addr 172.21.15.150:0/1976247200 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.896+0000 7f19a12ab640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 0x55d9229867e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 -- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979130 msgr2=0x55d9229862a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979130 0x55d9229862a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 -- 172.21.15.150:0/1976247200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 msgr2=0x55d9229867e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 0x55d9229867e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 -- 172.21.15.150:0/1976247200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9229f7e60 con 0x55d922982820 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a12ab640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 0x55d9229867e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a22ad640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1994000f80 tx=0x7f1994009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1aafdf5bb695a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19857fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994030040 con 0x55d922982820 2024-03-18T16:08:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8527+0+0 (secure 0 0 0) 0x7f199401a040 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1994011c50 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1994011e00 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a69da740 10 client.16146 fetch_fsmap learned FSMap version 31 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9229f9150 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.897+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1994011c50 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f19940051e0 con 0x55d922982820 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 10 client.16146 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 20 client.16146 populate_metadata read hostname 'smithi150' 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9229f9790 0x55d9229fbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 4 client.16146 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9228e52d0 con 0x55d9229f9790 2024-03-18T16:08:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a1aac640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9229f9790 0x55d9229fbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19857fa640 10 client.16146 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.898+0000 7f19a1aac640 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9229f9790 0x55d9229fbbb0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8312668fcdcf65f8 server_cookie=4df1301c7357d449 in_seq=0 out_seq=0 2024-03-18T16:08:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19857fa640 1 -- 172.21.15.150:0/1976247200 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9228e52d0 con 0x55d9229f9790 2024-03-18T16:08:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19857fa640 10 client.16146 handle_command_reply: tid=0 2024-03-18T16:08:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 client.16146 shutdown 2024-03-18T16:08:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 msgr2=0x55d922986d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1994000f80 tx=0x7f1994009500 comp rx=0 tx=0).stop 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 shutdown_connections 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55d9229f9790 0x55d9229fbbb0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d922982820 0x55d922986d20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d922979a50 0x55d9229867e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 --2- 172.21.15.150:0/1976247200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d922979130 0x55d9229862a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19857fa640 0 client.16146 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 >> 172.21.15.150:0/1976247200 conn(0x55d922973a50 msgr2=0x55d922972320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 shutdown_connections 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 1 -- 172.21.15.150:0/1976247200 wait complete. 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.909+0000 7f19a69da740 20 client.16146 trim_cache size 0 max 16384 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:08:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:08:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:08:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:08:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 msgr2=0x7f19880a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 0x7f19880a4f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1994005fd0 tx=0x7f1994009f10 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 msgr2=0x7f199c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1998006010 tx=0x7f199802cb30 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 shutdown_connections 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f19880a2a90 0x7f19880a4f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f199c105660 0x7f199c19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f199c1012d0 0x7f199c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 --2- 172.21.15.150:0/2381642862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c100930 0x7f199c198020 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 >> 172.21.15.150:0/2381642862 conn(0x7f199c0774b0 msgr2=0x7f199c06a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 shutdown_connections 2024-03-18T16:08:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:08:52.910+0000 7f19a3d10640 1 -- 172.21.15.150:0/2381642862 wait complete. 2024-03-18T16:08:53.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 894020 10.49 MB/sec execute 338 sec latency 41.640 ms 2024-03-18T16:08:53.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:53 smithi191 ceph-mon[36784]: pgmap v273: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:53 smithi156 ceph-mon[35919]: pgmap v273: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:53 smithi150 ceph-mon[31946]: pgmap v273: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:54.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 895636 10.48 MB/sec execute 339 sec latency 24.359 ms 2024-03-18T16:08:55.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 897667 10.49 MB/sec execute 340 sec latency 23.438 ms 2024-03-18T16:08:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:55 smithi191 ceph-mon[36784]: pgmap v274: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:55 smithi156 ceph-mon[35919]: pgmap v274: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:55 smithi150 ceph-mon[31946]: pgmap v274: 97 pgs: 97 active+clean; 530 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-03-18T16:08:56.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 899387 10.48 MB/sec execute 341 sec latency 13.558 ms 2024-03-18T16:08:57.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 901209 10.48 MB/sec execute 342 sec latency 77.129 ms 2024-03-18T16:08:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:57 smithi191 ceph-mon[36784]: pgmap v275: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 274 op/s 2024-03-18T16:08:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:57 smithi156 ceph-mon[35919]: pgmap v275: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 274 op/s 2024-03-18T16:08:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:57 smithi150 ceph-mon[31946]: pgmap v275: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 274 op/s 2024-03-18T16:08:58.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 902852 10.48 MB/sec execute 343 sec latency 18.273 ms 2024-03-18T16:08:59.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 905084 10.48 MB/sec execute 344 sec latency 18.833 ms 2024-03-18T16:08:59.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:08:59 smithi191 ceph-mon[36784]: pgmap v276: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-18T16:09:00.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:08:59 smithi156 ceph-mon[35919]: pgmap v276: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-18T16:09:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:08:59 smithi150 ceph-mon[31946]: pgmap v276: 97 pgs: 97 active+clean; 538 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-18T16:09:00.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 906542 10.47 MB/sec execute 345 sec latency 49.067 ms 2024-03-18T16:09:01.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 907788 10.46 MB/sec execute 346 sec latency 37.565 ms 2024-03-18T16:09:01.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:01 smithi191 ceph-mon[36784]: pgmap v277: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-18T16:09:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:01 smithi156 ceph-mon[35919]: pgmap v277: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-18T16:09:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:01 smithi150 ceph-mon[31946]: pgmap v277: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 269 op/s 2024-03-18T16:09:02.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 909427 10.46 MB/sec execute 347 sec latency 80.824 ms 2024-03-18T16:09:03.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 911039 10.45 MB/sec execute 348 sec latency 104.193 ms 2024-03-18T16:09:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:03 smithi191 ceph-mon[36784]: pgmap v278: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-03-18T16:09:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:03 smithi191 ceph-mon[36784]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:09:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:03 smithi156 ceph-mon[35919]: pgmap v278: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-03-18T16:09:04.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:03 smithi156 ceph-mon[35919]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:09:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:03 smithi150 ceph-mon[31946]: pgmap v278: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 182 op/s 2024-03-18T16:09:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:03 smithi150 ceph-mon[31946]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:09:04.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 912977 10.46 MB/sec execute 349 sec latency 19.897 ms 2024-03-18T16:09:05.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 914800 10.45 MB/sec execute 350 sec latency 29.885 ms 2024-03-18T16:09:05.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:05 smithi191 ceph-mon[36784]: pgmap v280: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 218 op/s 2024-03-18T16:09:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 -- 172.21.15.150:0/3069925192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 msgr2=0x7ff49010d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 --2- 172.21.15.150:0/3069925192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff49010d7a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff488009cd0 tx=0x7ff48802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 -- 172.21.15.150:0/3069925192 shutdown_connections 2024-03-18T16:09:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 --2- 172.21.15.150:0/3069925192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff49010d7a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 --2- 172.21.15.150:0/3069925192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 --2- 172.21.15.150:0/3069925192 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 0x7ff490106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 -- 172.21.15.150:0/3069925192 >> 172.21.15.150:0/3069925192 conn(0x7ff490076c70 msgr2=0x7ff4900fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 -- 172.21.15.150:0/3069925192 shutdown_connections 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.002+0000 7ff496619640 1 -- 172.21.15.150:0/3069925192 wait complete. 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 Processor -- start 2024-03-18T16:09:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 -- start start 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 0x7ff4901a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff490110cf0 con 0x7ff4901017b0 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff490110b70 con 0x7ff490100df0 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff496619640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff490110e70 con 0x7ff4901068e0 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35242/0 (socket says 172.21.15.150:35242) 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff48f7fe640 1 -- 172.21.15.150:0/3230139035 learned_addr learned my addr 172.21.15.150:0/3230139035 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.003+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 0x7ff4901a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 -- 172.21.15.150:0/3230139035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 msgr2=0x7ff4901a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 0x7ff4901a40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 -- 172.21.15.150:0/3230139035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 msgr2=0x7ff490199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 -- 172.21.15.150:0/3230139035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4901a4630 con 0x7ff4901017b0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e9f0 tx=0x7ff48400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48d7fa640 1 -- 172.21.15.150:0/3230139035 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48400cd80 con 0x7ff4901017b0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4901a4800 con 0x7ff4901017b0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48d7fa640 1 -- 172.21.15.150:0/3230139035 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff484010910 con 0x7ff4901017b0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff48d7fa640 1 -- 172.21.15.150:0/3230139035 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484018990 con 0x7ff4901017b0 2024-03-18T16:09:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.004+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4901a4cf0 con 0x7ff4901017b0 2024-03-18T16:09:06.014 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.006+0000 7ff48d7fa640 1 -- 172.21.15.150:0/3230139035 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7ff484010ac0 con 0x7ff4901017b0 2024-03-18T16:09:06.014 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 Processor -- start 2024-03-18T16:09:06.014 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 -- start start 2024-03-18T16:09:06.014 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48d7fa640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 0x7ff4780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f2186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f21b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560a2f16fa40 con 0x560a2f20b4f0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a2f118920 con 0x560a2f218c30 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff499208740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560a2f16bae0 con 0x560a2f20abd0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 0x7ff4780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48d7fa640 1 -- 172.21.15.150:0/3230139035 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7ff4840cf180 con 0x7ff4901017b0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:37648/0 (socket says 172.21.15.150:37648) 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 -- 172.21.15.150:0/1997646492 learned_addr learned my addr 172.21.15.150:0/1997646492 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f2186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff494bb6640 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f21b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 -- 172.21.15.150:0/1997646492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 msgr2=0x560a2f21b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f21b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 -- 172.21.15.150:0/1997646492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 msgr2=0x560a2f2186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f2186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48ffff640 1 -- 172.21.15.150:0/1997646492 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2f130460 con 0x560a2f20abd0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.007+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f2186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 0x7ff4780a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff488005cc0 tx=0x7ff48802ce60 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff48ffff640 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4800114f0 tx=0x7ff4800119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1932a0d031181af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff4767fc640 1 -- 172.21.15.150:0/1997646492 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48001e070 con 0x560a2f20abd0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff4767fc640 1 -- 172.21.15.150:0/1997646492 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48000faa0 con 0x560a2f20abd0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff4767fc640 1 -- 172.21.15.150:0/1997646492 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480019ba0 con 0x560a2f20abd0 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff499208740 1 -- 172.21.15.150:0/1997646492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 msgr2=0x560a2f20afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.008+0000 7ff499208740 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4800114f0 tx=0x7ff4800119b0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- 172.21.15.150:0/1997646492 shutdown_connections 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f21b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f2186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- 172.21.15.150:0/1997646492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f20afb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- 172.21.15.150:0/1997646492 >> 172.21.15.150:0/1997646492 conn(0x560a2f205a50 msgr2=0x560a2f205e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- 172.21.15.150:0/1997646492 shutdown_connections 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- 172.21.15.150:0/1997646492 wait complete. 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- start start 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 Processor -- start 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f20abd0 0x560a2f215c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f2161c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560a2f21b860 con 0x560a2f218c30 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560a2f12bd70 con 0x560a2f20abd0 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560a2f21b9e0 con 0x560a2f20b4f0 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff499208740 10 client.0 _ll_register_callbacks cb 0x560a2ee83a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff494bb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.017 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff494bb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35254/0 (socket says 172.21.15.150:35254) 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.009+0000 7ff494bb6640 1 -- 172.21.15.150:0/2057870349 learned_addr learned my addr 172.21.15.150:0/2057870349 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff48ffff640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f20abd0 0x560a2f215c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f2161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 -- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 msgr2=0x560a2f2161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f2161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 -- 172.21.15.150:0/2057870349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f20abd0 msgr2=0x560a2f215c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f20abd0 0x560a2f215c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 -- 172.21.15.150:0/2057870349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2ee84210 con 0x560a2f218c30 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff494bb6640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff488036000 tx=0x7ff48802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce56070b2a7c0e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488051060 con 0x560a2f218c30 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff48000faa0 con 0x560a2f218c30 2024-03-18T16:09:06.018 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7ff488003b70 con 0x560a2f218c30 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff488039040 con 0x560a2f218c30 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff499208740 10 client.16158 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.010+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560a2ef50620 con 0x560a2f218c30 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48000faa0 con 0x560a2f218c30 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff4880344d0 con 0x560a2f218c30 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 10 client.16158 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 20 client.16158 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f228e50 0x560a2f22b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 4 client.16158 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a2f17cf80 con 0x560a2f228e50 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff48ffff640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f228e50 0x560a2f22b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff46f7fe640 10 client.16158 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.011+0000 7ff48ffff640 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f228e50 0x560a2f22b240 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8922f86c896ee9a server_cookie=ee7883fd9a64a686 in_seq=0 out_seq=0 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff46f7fe640 1 -- 172.21.15.150:0/2057870349 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a2f17cf80 con 0x560a2f228e50 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff46f7fe640 10 client.16158 handle_command_reply: tid=0 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 client.16158 shutdown 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 msgr2=0x560a2f216700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff488036000 tx=0x7ff48802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 shutdown_connections 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f228e50 0x560a2f22b240 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f218c30 0x560a2f216700 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff46f7fe640 0 client.16158 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f2161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 --2- 172.21.15.150:0/2057870349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f20abd0 0x560a2f215c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 >> 172.21.15.150:0/2057870349 conn(0x560a2f205a50 msgr2=0x560a2f2077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 shutdown_connections 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.012+0000 7ff499208740 1 -- 172.21.15.150:0/2057870349 wait complete. 2024-03-18T16:09:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.013+0000 7ff499208740 20 client.16158 trim_cache size 0 max 16384 2024-03-18T16:09:06.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:05 smithi156 ceph-mon[35919]: pgmap v280: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 218 op/s 2024-03-18T16:09:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 Processor -- start 2024-03-18T16:09:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 -- start start 2024-03-18T16:09:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f224640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f216350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.028+0000 7ff499208740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901adc20 con 0x560a2f20abd0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff499208740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901ab4f0 con 0x560a2f218c30 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff499208740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901a9360 con 0x560a2f20b4f0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f224640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35270/0 (socket says 172.21.15.150:35270) 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 -- 172.21.15.150:0/3860398154 learned_addr learned my addr 172.21.15.150:0/3860398154 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f216350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 -- 172.21.15.150:0/3860398154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 msgr2=0x560a2f224640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f224640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 -- 172.21.15.150:0/3860398154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 msgr2=0x560a2f216350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f216350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 -- 172.21.15.150:0/3860398154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2f20bb20 con 0x560a2f20abd0 2024-03-18T16:09:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f224640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f216350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff4800114c0 tx=0x7ff4800096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e1a807b43a7cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.029+0000 7ff475ffb640 1 -- 172.21.15.150:0/3860398154 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48002f070 con 0x560a2f20abd0 2024-03-18T16:09:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff475ffb640 1 -- 172.21.15.150:0/3860398154 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48000faa0 con 0x560a2f20abd0 2024-03-18T16:09:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff475ffb640 1 -- 172.21.15.150:0/3860398154 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480029030 con 0x560a2f20abd0 2024-03-18T16:09:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 -- 172.21.15.150:0/3860398154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 msgr2=0x560a2f2183b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff4800114c0 tx=0x7ff4800096e0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 -- 172.21.15.150:0/3860398154 shutdown_connections 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f216350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20b4f0 0x560a2f224640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 --2- 172.21.15.150:0/3860398154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20abd0 0x560a2f2183b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 -- 172.21.15.150:0/3860398154 >> 172.21.15.150:0/3860398154 conn(0x560a2f205a50 msgr2=0x560a2f215c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 -- 172.21.15.150:0/3860398154 shutdown_connections 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.030+0000 7ff499208740 1 -- 172.21.15.150:0/3860398154 wait complete. 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 -- start start 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 Processor -- start 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f2134b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901a99b0 con 0x560a2f20b4f0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901aae20 con 0x560a2f218c30 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff4901aa080 con 0x560a2f20abd0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff499208740 10 client.0 _ll_register_callbacks cb 0x7ff484005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff494bb6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f2134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:37684/0 (socket says 172.21.15.150:37684) 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff48f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35272/0 (socket says 172.21.15.150:35272) 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.031+0000 7ff48ffff640 1 -- 172.21.15.150:0/3527505947 learned_addr learned my addr 172.21.15.150:0/3527505947 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 -- 172.21.15.150:0/3527505947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 msgr2=0x560a2f2134b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f2134b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 -- 172.21.15.150:0/3527505947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 msgr2=0x560a2f212f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 -- 172.21.15.150:0/3527505947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2f21b020 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff494bb6640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f2134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff48ffff640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff48000a010 tx=0x7ff480005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e383750088e2cf27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff46f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48002f070 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7ff48000fcd0 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff48000faa0 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff480029070 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff499208740 10 client.25987 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560a2f28f4b0 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.032+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48000faa0 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff48003b070 con 0x560a2f20abd0 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 10 client.25987 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 20 client.25987 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f28f920 0x560a2f291d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 4 client.25987 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560a2f1772d0 con 0x560a2f28f920 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f28f920 0x560a2f291d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff46f7fe640 10 client.25987 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.033+0000 7ff48f7fe640 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f28f920 0x560a2f291d10 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efd365a28021509c server_cookie=40300d8b24d3539e in_seq=0 out_seq=0 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff46f7fe640 1 -- 172.21.15.150:0/3527505947 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560a2f1772d0 con 0x560a2f28f920 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff46f7fe640 10 client.25987 handle_command_reply: tid=0 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 client.25987 shutdown 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 msgr2=0x560a2f212a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff48000a010 tx=0x7ff480005fd0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 shutdown_connections 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560a2f28f920 0x560a2f291d10 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a2f218c30 0x560a2f2134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a2f20b4f0 0x560a2f212f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 --2- 172.21.15.150:0/3527505947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560a2f20abd0 0x560a2f212a30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff46f7fe640 0 client.25987 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 >> 172.21.15.150:0/3527505947 conn(0x560a2f205a50 msgr2=0x560a2f204bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 shutdown_connections 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 1 -- 172.21.15.150:0/3527505947 wait complete. 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.045+0000 7ff499208740 20 client.25987 trim_cache size 0 max 16384 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:09:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:09:06.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 msgr2=0x7ff4780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 0x7ff4780a4ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff488005cc0 tx=0x7ff48802ce60 comp rx=0 tx=0).stop 2024-03-18T16:09:06.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 msgr2=0x7ff4901995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.049 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e9f0 tx=0x7ff48400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 shutdown_connections 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff4780a2a10 0x7ff4780a4ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4901068e0 0x7ff4901a40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4901017b0 0x7ff4901995b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 --2- 172.21.15.150:0/3230139035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff490100df0 0x7ff490199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 >> 172.21.15.150:0/3230139035 conn(0x7ff490076c70 msgr2=0x7ff4901035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 shutdown_connections 2024-03-18T16:09:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:06.046+0000 7ff496619640 1 -- 172.21.15.150:0/3230139035 wait complete. 2024-03-18T16:09:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:05 smithi150 ceph-mon[31946]: pgmap v280: 97 pgs: 97 active+clean; 547 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 218 op/s 2024-03-18T16:09:06.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 916549 10.45 MB/sec execute 351 sec latency 10.627 ms 2024-03-18T16:09:07.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 917479 10.48 MB/sec execute 352 sec latency 91.438 ms 2024-03-18T16:09:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:07 smithi191 ceph-mon[36784]: pgmap v281: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:07 smithi156 ceph-mon[35919]: pgmap v281: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:07 smithi150 ceph-mon[31946]: pgmap v281: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:08.336 INFO:tasks.workunit.client.0.smithi150.stdout: 1 919127 10.47 MB/sec execute 353 sec latency 74.435 ms 2024-03-18T16:09:09.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 920964 10.48 MB/sec execute 354 sec latency 15.998 ms 2024-03-18T16:09:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:09 smithi191 ceph-mon[36784]: pgmap v282: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:09 smithi191 ceph-mon[36784]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:09:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:09 smithi156 ceph-mon[35919]: pgmap v282: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:09 smithi156 ceph-mon[35919]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:09:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:09 smithi150 ceph-mon[31946]: pgmap v282: 97 pgs: 97 active+clean; 556 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 254 op/s 2024-03-18T16:09:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:09 smithi150 ceph-mon[31946]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:09:10.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 923023 10.48 MB/sec execute 355 sec latency 18.625 ms 2024-03-18T16:09:11.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 924713 10.47 MB/sec execute 356 sec latency 20.680 ms 2024-03-18T16:09:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:11 smithi191 ceph-mon[36784]: pgmap v284: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 318 op/s 2024-03-18T16:09:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:11 smithi156 ceph-mon[35919]: pgmap v284: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 318 op/s 2024-03-18T16:09:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:11 smithi150 ceph-mon[31946]: pgmap v284: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 318 op/s 2024-03-18T16:09:12.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 926756 10.48 MB/sec execute 357 sec latency 29.698 ms 2024-03-18T16:09:13.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 928126 10.47 MB/sec execute 358 sec latency 73.376 ms 2024-03-18T16:09:13.962 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:13 smithi191 ceph-mon[36784]: pgmap v285: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-03-18T16:09:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:13 smithi156 ceph-mon[35919]: pgmap v285: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-03-18T16:09:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:13 smithi150 ceph-mon[31946]: pgmap v285: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 256 op/s 2024-03-18T16:09:14.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 930118 10.47 MB/sec execute 359 sec latency 17.985 ms 2024-03-18T16:09:15.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 932062 10.47 MB/sec execute 360 sec latency 13.406 ms 2024-03-18T16:09:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:15 smithi191 ceph-mon[36784]: pgmap v286: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-03-18T16:09:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:15 smithi156 ceph-mon[35919]: pgmap v286: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-03-18T16:09:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:15 smithi150 ceph-mon[31946]: pgmap v286: 97 pgs: 97 active+clean; 566 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-03-18T16:09:16.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 934402 10.48 MB/sec execute 361 sec latency 18.204 ms 2024-03-18T16:09:17.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 935888 10.47 MB/sec execute 362 sec latency 39.173 ms 2024-03-18T16:09:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:17 smithi191 ceph-mon[36784]: pgmap v287: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:17 smithi156 ceph-mon[35919]: pgmap v287: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:17 smithi150 ceph-mon[31946]: pgmap v287: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:18.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 937808 10.47 MB/sec execute 363 sec latency 99.330 ms 2024-03-18T16:09:18.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:18 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:19.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:18 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:19.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:18 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 -- 172.21.15.150:0/3704457759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1077c0 msgr2=0x7f49fc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/3704457759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1077c0 0x7f49fc107ba0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009930 tx=0x7f49f802bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 -- 172.21.15.150:0/3704457759 shutdown_connections 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/3704457759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc102170 0x7f49fc10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/3704457759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc1017b0 0x7f49fc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/3704457759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1077c0 0x7f49fc107ba0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 -- 172.21.15.150:0/3704457759 >> 172.21.15.150:0/3704457759 conn(0x7f49fc0fd2c0 msgr2=0x7f49fc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 -- 172.21.15.150:0/3704457759 shutdown_connections 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.142+0000 7f4a03ca1640 1 -- 172.21.15.150:0/3704457759 wait complete. 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 Processor -- start 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 -- start start 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc102170 0x7f49fc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 0x7f49fc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc111b90 con 0x7f49fc1017b0 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc111a10 con 0x7f49fc1077c0 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a03ca1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc111d10 con 0x7f49fc102170 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48186/0 (socket says 172.21.15.150:48186) 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 -- 172.21.15.150:0/4227560446 learned_addr learned my addr 172.21.15.150:0/4227560446 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 0x7f49fc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a0123c640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc102170 0x7f49fc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 -- 172.21.15.150:0/4227560446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc102170 msgr2=0x7f49fc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc102170 0x7f49fc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 -- 172.21.15.150:0/4227560446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 msgr2=0x7f49fc19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.143+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 0x7f49fc19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f4a01a3d640 1 -- 172.21.15.150:0/4227560446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc19d020 con 0x7f49fc1017b0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 0x7f49fc19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009230 tx=0x7f49f802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f49f2ffd640 1 -- 172.21.15.150:0/4227560446 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8041070 con 0x7f49fc1017b0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f49f2ffd640 1 -- 172.21.15.150:0/4227560446 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f8004450 con 0x7f49fc1017b0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f49f2ffd640 1 -- 172.21.15.150:0/4227560446 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8003e50 con 0x7f49fc1017b0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49fc19d2b0 con 0x7f49fc1017b0 2024-03-18T16:09:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.144+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49fc1a4ba0 con 0x7f49fc1017b0 2024-03-18T16:09:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f49f2ffd640 1 -- 172.21.15.150:0/4227560446 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f49f8044070 con 0x7f49fc1017b0 2024-03-18T16:09:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 Processor -- start 2024-03-18T16:09:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 -- start start 2024-03-18T16:09:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6aa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562c8c60ea40 con 0x562c8c6aaa50 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562c8c5b7920 con 0x562c8c6aa130 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.147+0000 7f4a0696a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562c8c60aae0 con 0x562c8c6b3820 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f49f2ffd640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 0x7f49e80a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48200/0 (socket says 172.21.15.150:48200) 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:41932/0 (socket says 172.21.15.150:41932) 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a01a3d640 1 -- 172.21.15.150:0/891482831 learned_addr learned my addr 172.21.15.150:0/891482831 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 0x7f49e80a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f49f2ffd640 1 -- 172.21.15.150:0/4227560446 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f49f80f0620 con 0x7f49fc1017b0 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 -- 172.21.15.150:0/891482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 msgr2=0x562c8c6aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6aa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 -- 172.21.15.150:0/891482831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 msgr2=0x562c8c6b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 -- 172.21.15.150:0/891482831 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8c5cf460 con 0x562c8c6b3820 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 0x7f49e80a5070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f49f40099e0 tx=0x7f49f4009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f4a0123c640 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6b5c10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009bf0 tx=0x7f49ec011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d10b85155318d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.148+0000 7f49e7fff640 1 -- 172.21.15.150:0/891482831 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec004050 con 0x562c8c6b3820 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f49e7fff640 1 -- 172.21.15.150:0/891482831 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49ec005500 con 0x562c8c6b3820 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f49e7fff640 1 -- 172.21.15.150:0/891482831 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec004e00 con 0x562c8c6b3820 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 -- 172.21.15.150:0/891482831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 msgr2=0x562c8c6b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6b5c10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009bf0 tx=0x7f49ec011810 comp rx=0 tx=0).stop 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 -- 172.21.15.150:0/891482831 shutdown_connections 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6b5c10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 --2- 172.21.15.150:0/891482831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 -- 172.21.15.150:0/891482831 >> 172.21.15.150:0/891482831 conn(0x562c8c6a4a50 msgr2=0x562c8c6a4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 -- 172.21.15.150:0/891482831 shutdown_connections 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.149+0000 7f4a0696a740 1 -- 172.21.15.150:0/891482831 wait complete. 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 -- start start 2024-03-18T16:09:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 Processor -- start 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 0x562c8c6b0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x562c8c6b6450 con 0x562c8c6aaa50 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562c8c5cad70 con 0x562c8c6b3820 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x562c8c6b65d0 con 0x562c8c6aa130 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a0696a740 10 client.0 _ll_register_callbacks cb 0x562c8c322a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.150+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 0x562c8c6b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 0x562c8c6b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:33056/0 (socket says 172.21.15.150:33056) 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48216/0 (socket says 172.21.15.150:48216) 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a01a3d640 1 -- 172.21.15.150:0/4280987263 learned_addr learned my addr 172.21.15.150:0/4280987263 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:41944/0 (socket says 172.21.15.150:41944) 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 -- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 msgr2=0x562c8c6b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 0x562c8c6b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 -- 172.21.15.150:0/4280987263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 msgr2=0x562c8c6b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 -- 172.21.15.150:0/4280987263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8c323210 con 0x562c8c6b3820 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0223e640 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011fd0 tx=0x7f49f4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f963dff1327b2242 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f49e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f400fd80 con 0x562c8c6b3820 2024-03-18T16:09:19.158 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f49f4004b20 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49ec005500 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f49f4004d20 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0696a740 10 client.25940 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.151+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c8c3ef620 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49ec005500 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f49f401d070 con 0x562c8c6b3820 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 10 client.25940 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 20 client.25940 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c6c3a10 0x562c8c6c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 4 client.25940 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c8c61bf80 con 0x562c8c6c3a10 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c6c3a10 0x562c8c6c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c6c3a10 0x562c8c6c5e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c7e4998bc2cab3c server_cookie=e046223137c8ad59 in_seq=0 out_seq=0 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.152+0000 7f49e57fa640 10 client.25940 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:19.159 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f49e57fa640 1 -- 172.21.15.150:0/4280987263 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c8c61bf80 con 0x562c8c6c3a10 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f49e57fa640 10 client.25940 handle_command_reply: tid=0 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 client.25940 shutdown 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 msgr2=0x562c8c6b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011fd0 tx=0x7f49f4004a40 comp rx=0 tx=0).stop 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 shutdown_connections 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c6c3a10 0x562c8c6c5e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6b3820 0x562c8c6b12c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f49e57fa640 0 client.25940 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c6b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 --2- 172.21.15.150:0/4280987263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aa130 0x562c8c6b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 >> 172.21.15.150:0/4280987263 conn(0x562c8c6a4a50 msgr2=0x562c8c6a67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.153+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 shutdown_connections 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.154+0000 7f4a0696a740 1 -- 172.21.15.150:0/4280987263 wait complete. 2024-03-18T16:09:19.160 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.154+0000 7f4a0696a740 20 client.25940 trim_cache size 0 max 16384 2024-03-18T16:09:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.169+0000 7f4a0696a740 1 Processor -- start 2024-03-18T16:09:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.169+0000 7f4a0696a740 1 -- start start 2024-03-18T16:09:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1afde0 con 0x562c8c6aaa50 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1ad6b0 con 0x562c8c6aa130 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0696a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1ab520 con 0x562c8c6b3820 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48228/0 (socket says 172.21.15.150:48228) 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 -- 172.21.15.150:0/176659080 learned_addr learned my addr 172.21.15.150:0/176659080 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0223e640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 -- 172.21.15.150:0/176659080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 msgr2=0x562c8c6bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6bf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 -- 172.21.15.150:0/176659080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 msgr2=0x562c8c6b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 -- 172.21.15.150:0/176659080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8c6af810 con 0x562c8c6aaa50 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0223e640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f4a0123c640 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000c00 tx=0x7f49ec009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11330c1917f29d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.170+0000 7f49e77fe640 1 -- 172.21.15.150:0/176659080 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0052b0 con 0x562c8c6aaa50 2024-03-18T16:09:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f49e77fe640 1 -- 172.21.15.150:0/176659080 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49ec028070 con 0x562c8c6aaa50 2024-03-18T16:09:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f49e77fe640 1 -- 172.21.15.150:0/176659080 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec019690 con 0x562c8c6aaa50 2024-03-18T16:09:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 -- 172.21.15.150:0/176659080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 msgr2=0x562c8c69fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000c00 tx=0x7f49ec009ce0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 -- 172.21.15.150:0/176659080 shutdown_connections 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6b3820 0x562c8c6bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6aaa50 0x562c8c69fa40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 --2- 172.21.15.150:0/176659080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 -- 172.21.15.150:0/176659080 >> 172.21.15.150:0/176659080 conn(0x562c8c6a4a50 msgr2=0x562c8c6a6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.171+0000 7f4a0696a740 1 -- 172.21.15.150:0/176659080 shutdown_connections 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 -- 172.21.15.150:0/176659080 wait complete. 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 -- start start 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 Processor -- start 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6bb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aaa50 0x562c8c6b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1abb70 con 0x562c8c6b3820 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1acfe0 con 0x562c8c6aa130 2024-03-18T16:09:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc1ac240 con 0x562c8c6aaa50 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0696a740 10 client.0 _ll_register_callbacks cb 0x7f49f4011bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0223e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48244/0 (socket says 172.21.15.150:48244) 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a01a3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0223e640 1 -- 172.21.15.150:0/2757479690 learned_addr learned my addr 172.21.15.150:0/2757479690 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.172+0000 7f4a0123c640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aaa50 0x562c8c6b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 -- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aaa50 msgr2=0x562c8c6b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aaa50 0x562c8c6b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 -- 172.21.15.150:0/2757479690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 msgr2=0x562c8c6bb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6bb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 -- 172.21.15.150:0/2757479690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8c6b9270 con 0x562c8c6b3820 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6bb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0223e640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011c80 tx=0x7f49f4011910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61fe0c9b541c485 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f49e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f401f720 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f49f4011dd0 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49ec028070 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f49f401d760 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0696a740 10 client.16182 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.173+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c8c72a9c0 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49ec028070 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f49f4004380 con 0x562c8c6b3820 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 10 client.16182 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 20 client.16182 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c72b0d0 0x562c8c72d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 4 client.16182 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c8c6162d0 con 0x562c8c72b0d0 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c72b0d0 0x562c8c72d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f4a01a3d640 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c72b0d0 0x562c8c72d4f0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0a7f65ca3a1e258 server_cookie=88db8af7679529f8 in_seq=0 out_seq=0 2024-03-18T16:09:19.180 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.174+0000 7f49e57fa640 10 client.16182 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f49e57fa640 1 -- 172.21.15.150:0/2757479690 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c8c6162d0 con 0x562c8c72b0d0 2024-03-18T16:09:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f49e57fa640 10 client.16182 handle_command_reply: tid=0 2024-03-18T16:09:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 client.16182 shutdown 2024-03-18T16:09:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 msgr2=0x562c8c6b8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011c80 tx=0x7f49f4011910 comp rx=0 tx=0).stop 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 shutdown_connections 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x562c8c72b0d0 0x562c8c72d4f0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562c8c6b3820 0x562c8c6b8d30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x562c8c6aaa50 0x562c8c6b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f49e57fa640 0 client.16182 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 --2- 172.21.15.150:0/2757479690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562c8c6aa130 0x562c8c6bb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 >> 172.21.15.150:0/2757479690 conn(0x562c8c6a4a50 msgr2=0x562c8c6a5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 shutdown_connections 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.185+0000 7f4a0696a740 1 -- 172.21.15.150:0/2757479690 wait complete. 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a0696a740 20 client.16182 trim_cache size 0 max 16384 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:09:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:09:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:09:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:09:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 msgr2=0x7f49e80a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 0x7f49e80a5070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f49f40099e0 tx=0x7f49f4009500 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 msgr2=0x7f49fc198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009230 tx=0x7f49f802cb30 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 shutdown_connections 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f49e80a2bb0 0x7f49e80a5070 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49fc1077c0 0x7f49fc19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49fc102170 0x7f49fc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 --2- 172.21.15.150:0/4227560446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49fc1017b0 0x7f49fc198030 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 >> 172.21.15.150:0/4227560446 conn(0x7f49fc0fd2c0 msgr2=0x7f49fc0fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 shutdown_connections 2024-03-18T16:09:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:19.186+0000 7f4a03ca1640 1 -- 172.21.15.150:0/4227560446 wait complete. 2024-03-18T16:09:19.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 939264 10.46 MB/sec execute 364 sec latency 19.373 ms 2024-03-18T16:09:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:19 smithi191 ceph-mon[36784]: pgmap v288: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:19 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:19 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:19 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:09:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:19 smithi156 ceph-mon[35919]: pgmap v288: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:19 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:20.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:19 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:20.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:19 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:09:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:19 smithi150 ceph-mon[31946]: pgmap v288: 97 pgs: 97 active+clean; 536 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 208 op/s 2024-03-18T16:09:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:19 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:19 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:19 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:09:20.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 941376 10.46 MB/sec execute 365 sec latency 18.956 ms 2024-03-18T16:09:21.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 943015 10.46 MB/sec execute 366 sec latency 21.138 ms 2024-03-18T16:09:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:21 smithi191 ceph-mon[36784]: pgmap v289: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:09:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:21 smithi156 ceph-mon[35919]: pgmap v289: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:09:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:21 smithi150 ceph-mon[31946]: pgmap v289: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:09:22.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 945212 10.47 MB/sec execute 367 sec latency 55.850 ms 2024-03-18T16:09:23.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 946384 10.45 MB/sec execute 368 sec latency 50.085 ms 2024-03-18T16:09:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:23 smithi156 ceph-mon[35919]: pgmap v290: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:23 smithi150 ceph-mon[31946]: pgmap v290: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:24.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:23 smithi191 ceph-mon[36784]: pgmap v290: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:24.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 948651 10.46 MB/sec execute 369 sec latency 19.143 ms 2024-03-18T16:09:25.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 950211 10.45 MB/sec execute 370 sec latency 21.753 ms 2024-03-18T16:09:26.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:25 smithi156 ceph-mon[35919]: pgmap v291: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:26.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:25 smithi150 ceph-mon[31946]: pgmap v291: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:26.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:25 smithi191 ceph-mon[36784]: pgmap v291: 97 pgs: 97 active+clean; 544 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-03-18T16:09:26.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 952475 10.46 MB/sec execute 371 sec latency 17.121 ms 2024-03-18T16:09:27.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 954111 10.45 MB/sec execute 372 sec latency 19.545 ms 2024-03-18T16:09:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:27 smithi156 ceph-mon[35919]: pgmap v292: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 260 op/s 2024-03-18T16:09:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:27 smithi150 ceph-mon[31946]: pgmap v292: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 260 op/s 2024-03-18T16:09:28.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:27 smithi191 ceph-mon[36784]: pgmap v292: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.7 MiB/s wr, 260 op/s 2024-03-18T16:09:28.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 956035 10.45 MB/sec execute 373 sec latency 83.867 ms 2024-03-18T16:09:29.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 957393 10.44 MB/sec execute 374 sec latency 52.413 ms 2024-03-18T16:09:30.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:29 smithi156 ceph-mon[35919]: pgmap v293: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:09:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:29 smithi150 ceph-mon[31946]: pgmap v293: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:09:30.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:29 smithi191 ceph-mon[36784]: pgmap v293: 97 pgs: 97 active+clean; 553 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:09:30.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 959542 10.45 MB/sec execute 375 sec latency 17.042 ms 2024-03-18T16:09:31.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 961111 10.44 MB/sec execute 376 sec latency 17.351 ms 2024-03-18T16:09:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:31 smithi156 ceph-mon[35919]: pgmap v294: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 260 op/s 2024-03-18T16:09:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:31 smithi150 ceph-mon[31946]: pgmap v294: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 260 op/s 2024-03-18T16:09:32.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:31 smithi191 ceph-mon[36784]: pgmap v294: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 260 op/s 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 -- 172.21.15.150:0/46452062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0660100930 msgr2=0x7f0660100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 --2- 172.21.15.150:0/46452062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0660100930 0x7f0660100d90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0648009900 tx=0x7f064802bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 -- 172.21.15.150:0/46452062 shutdown_connections 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 --2- 172.21.15.150:0/46452062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06601012d0 0x7f066010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 --2- 172.21.15.150:0/46452062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0660100930 0x7f0660100d90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 --2- 172.21.15.150:0/46452062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 0x7f0660105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 -- 172.21.15.150:0/46452062 >> 172.21.15.150:0/46452062 conn(0x7f06600774b0 msgr2=0x7f06600778c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 -- 172.21.15.150:0/46452062 shutdown_connections 2024-03-18T16:09:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.278+0000 7f06648e8640 1 -- 172.21.15.150:0/46452062 wait complete. 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 Processor -- start 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 -- start start 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 0x7f06600fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0660110b70 con 0x7f06601012d0 2024-03-18T16:09:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06601109f0 con 0x7f0660100930 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.279+0000 7f06648e8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0660110cf0 con 0x7f0660105660 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0655d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065ed9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 0x7f06600fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0655d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45258/0 (socket says 172.21.15.150:45258) 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33234/0 (socket says 172.21.15.150:33234) 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0655d9b640 1 -- 172.21.15.150:0/3164716077 learned_addr learned my addr 172.21.15.150:0/3164716077 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 -- 172.21.15.150:0/3164716077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 msgr2=0x7f06600fedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 0x7f06600fedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 -- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 msgr2=0x7f06600fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 -- 172.21.15.150:0/3164716077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06601a6500 con 0x7f0660100930 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0655d9b640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f065e59c640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f065000e9f0 tx=0x7f065000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0657fff640 1 -- 172.21.15.150:0/3164716077 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065000ccc0 con 0x7f0660100930 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06601a67f0 con 0x7f0660100930 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0657fff640 1 -- 172.21.15.150:0/3164716077 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0650010910 con 0x7f0660100930 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f0657fff640 1 -- 172.21.15.150:0/3164716077 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650018960 con 0x7f0660100930 2024-03-18T16:09:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.280+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06601a6d40 con 0x7f0660100930 2024-03-18T16:09:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.282+0000 7f0657fff640 1 -- 172.21.15.150:0/3164716077 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f0650010ac0 con 0x7f0660100930 2024-03-18T16:09:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 Processor -- start 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f0657fff640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 0x7f064c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 0x7f064c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f0657fff640 1 -- 172.21.15.150:0/3164716077 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f06500cf0f0 con 0x7f0660100930 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 -- start start 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590389a6820 0x5590389a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x559038901a40 con 0x55903899da50 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5590388aa920 con 0x55903899d130 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f06675a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5590388fdae0 con 0x5590389a6820 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f0655d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.293 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33246/0 (socket says 172.21.15.150:33246) 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f065e59c640 1 -- 172.21.15.150:0/2181622965 learned_addr learned my addr 172.21.15.150:0/2181622965 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.283+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 0x7f064c0a4ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0658005fd0 tx=0x7f0658009500 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065ed9d640 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590389a6820 0x5590389a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 -- 172.21.15.150:0/2181622965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590389a6820 msgr2=0x5590389a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590389a6820 0x5590389a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 -- 172.21.15.150:0/2181622965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 msgr2=0x5590389a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 -- 172.21.15.150:0/2181622965 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590388c2460 con 0x55903899d130 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f065e59c640 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f065009b830 tx=0x7f06500974a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fff44889fddb624 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f0654d99640 1 -- 172.21.15.150:0/2181622965 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06500ccb60 con 0x55903899d130 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f0654d99640 1 -- 172.21.15.150:0/2181622965 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0650097f10 con 0x55903899d130 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.284+0000 7f0654d99640 1 -- 172.21.15.150:0/2181622965 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065009afe0 con 0x55903899d130 2024-03-18T16:09:32.294 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- 172.21.15.150:0/2181622965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 msgr2=0x55903899d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f065009b830 tx=0x7f06500974a0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- 172.21.15.150:0/2181622965 shutdown_connections 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5590389a6820 0x5590389a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- 172.21.15.150:0/2181622965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x55903899d510 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- 172.21.15.150:0/2181622965 >> 172.21.15.150:0/2181622965 conn(0x559038997a50 msgr2=0x559038997e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- 172.21.15.150:0/2181622965 shutdown_connections 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- 172.21.15.150:0/2181622965 wait complete. 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- start start 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 Processor -- start 2024-03-18T16:09:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5590389a93f0 con 0x5590389a6820 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5590388bdd70 con 0x55903899d130 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.285+0000 7f06675a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5590389a9570 con 0x55903899da50 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f06675a9740 10 client.0 _ll_register_callbacks cb 0x559038615a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45270/0 (socket says 172.21.15.150:45270) 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33254/0 (socket says 172.21.15.150:33254) 2024-03-18T16:09:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 -- 172.21.15.150:0/3862108010 learned_addr learned my addr 172.21.15.150:0/3862108010 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f0655d9b640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 -- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 msgr2=0x5590389a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 -- 172.21.15.150:0/3862108010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 msgr2=0x5590389a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 -- 172.21.15.150:0/3862108010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559038616210 con 0x5590389a6820 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065e59c640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f065800ed00 tx=0x7f0658004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b83690212ecfe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f0645ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658028440 con 0x5590389a6820 2024-03-18T16:09:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f06580273e0 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0650097f10 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06580285e0 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f06675a9740 10 client.16194 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.286+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5590386e2620 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0650097f10 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0658026070 con 0x5590389a6820 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 10 client.16194 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 20 client.16194 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590389b6ab0 0x5590389b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 4 client.16194 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55903890ef80 con 0x5590389b6ab0 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f065e59c640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590389b6ab0 0x5590389b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f0645ffb640 10 client.16194 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.287+0000 7f065e59c640 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590389b6ab0 0x5590389b8ea0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91809b20dec52a0 server_cookie=244f7daeac4b487c in_seq=0 out_seq=0 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.288+0000 7f0645ffb640 1 -- 172.21.15.150:0/3862108010 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55903890ef80 con 0x5590389b6ab0 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.288+0000 7f0645ffb640 10 client.16194 handle_command_reply: tid=0 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.288+0000 7f06675a9740 1 client.16194 shutdown 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 msgr2=0x5590389a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f065800ed00 tx=0x7f0658004ab0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 shutdown_connections 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5590389b6ab0 0x5590389b8ea0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5590389a6820 0x5590389a4380 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 --2- 172.21.15.150:0/3862108010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55903899d130 0x5590389a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 >> 172.21.15.150:0/3862108010 conn(0x559038997a50 msgr2=0x5590389997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f0645ffb640 0 client.16194 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 shutdown_connections 2024-03-18T16:09:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 1 -- 172.21.15.150:0/3862108010 wait complete. 2024-03-18T16:09:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.289+0000 7f06675a9740 20 client.16194 trim_cache size 0 max 16384 2024-03-18T16:09:32.307 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 Processor -- start 2024-03-18T16:09:32.307 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 -- start start 2024-03-18T16:09:32.307 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389be7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06601b1f10 con 0x55903899d130 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06601af7e0 con 0x5590389a6820 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.304+0000 7f06675a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f06601ad650 con 0x55903899da50 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45286/0 (socket says 172.21.15.150:45286) 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 -- 172.21.15.150:0/2632057588 learned_addr learned my addr 172.21.15.150:0/2632057588 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f0655d9b640 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065ed9d640 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389be7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 -- 172.21.15.150:0/2632057588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 msgr2=0x5590389b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 -- 172.21.15.150:0/2632057588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 msgr2=0x5590389be7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389be7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 -- 172.21.15.150:0/2632057588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903899c8f0 con 0x55903899d130 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f065e59c640 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f06500cf480 tx=0x7f065000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1884f2c31eaa9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f0647fff640 1 -- 172.21.15.150:0/2632057588 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06500b4020 con 0x55903899d130 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f0647fff640 1 -- 172.21.15.150:0/2632057588 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06500860b0 con 0x55903899d130 2024-03-18T16:09:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.305+0000 7f0647fff640 1 -- 172.21.15.150:0/2632057588 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065009acf0 con 0x55903899d130 2024-03-18T16:09:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 -- 172.21.15.150:0/2632057588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 msgr2=0x5590389a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f06500cf480 tx=0x7f065000cf60 comp rx=0 tx=0).stop 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 -- 172.21.15.150:0/2632057588 shutdown_connections 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389be7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899da50 0x5590389b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 --2- 172.21.15.150:0/2632057588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899d130 0x5590389a6030 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 -- 172.21.15.150:0/2632057588 >> 172.21.15.150:0/2632057588 conn(0x559038997a50 msgr2=0x55903899ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 -- 172.21.15.150:0/2632057588 shutdown_connections 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.306+0000 7f06675a9740 1 -- 172.21.15.150:0/2632057588 wait complete. 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 -- start start 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 Processor -- start 2024-03-18T16:09:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899d130 0x5590389ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06601adca0 con 0x55903899da50 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06601af110 con 0x5590389a6820 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f06601ae370 con 0x55903899d130 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f06675a9740 10 client.0 _ll_register_callbacks cb 0x7f065800e4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45292/0 (socket says 172.21.15.150:45292) 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 -- 172.21.15.150:0/3359513255 learned_addr learned my addr 172.21.15.150:0/3359513255 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f065e59c640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899d130 0x5590389ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f065ed9d640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 -- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899d130 msgr2=0x5590389ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899d130 0x5590389ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 -- 172.21.15.150:0/3359513255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 msgr2=0x5590389aa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389aa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.307+0000 7f0655d9b640 1 -- 172.21.15.150:0/3359513255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590389aa670 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0655d9b640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0648006010 tx=0x7f064802cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c140ad917586b5a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648004f00 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7f06480050a0 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06500860b0 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0648055050 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 10 client.16200 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559038a1cbb0 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06500860b0 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0648050030 con 0x55903899da50 2024-03-18T16:09:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 10 client.16200 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 20 client.16200 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559038a1d3b0 0x559038a1f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 4 client.16200 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590389092d0 con 0x559038a1d3b0 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.308+0000 7f065e59c640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559038a1d3b0 0x559038a1f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.309+0000 7f0645ffb640 10 client.16200 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.309+0000 7f065e59c640 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559038a1d3b0 0x559038a1f7d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff4aa6107e924a45 server_cookie=83261cdce9cb7f4e in_seq=0 out_seq=0 2024-03-18T16:09:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f0645ffb640 1 -- 172.21.15.150:0/3359513255 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590389092d0 con 0x559038a1d3b0 2024-03-18T16:09:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f0645ffb640 10 client.16200 handle_command_reply: tid=0 2024-03-18T16:09:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 client.16200 shutdown 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 msgr2=0x5590389a9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0648006010 tx=0x7f064802cab0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 shutdown_connections 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x559038a1d3b0 0x559038a1f7d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5590389a6820 0x5590389aa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55903899da50 0x5590389a9b70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 --2- 172.21.15.150:0/3359513255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55903899d130 0x5590389ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f0645ffb640 0 client.16200 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 >> 172.21.15.150:0/3359513255 conn(0x559038997a50 msgr2=0x5590389a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 shutdown_connections 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 1 -- 172.21.15.150:0/3359513255 wait complete. 2024-03-18T16:09:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.320+0000 7f06675a9740 20 client.16200 trim_cache size 0 max 16384 2024-03-18T16:09:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:09:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:09:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:09:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 msgr2=0x7f064c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 0x7f064c0a4ed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0658005fd0 tx=0x7f0658009500 comp rx=0 tx=0).stop 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 msgr2=0x7f06600fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f065000e9f0 tx=0x7f065000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 shutdown_connections 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f064c0a2a10 0x7f064c0a4ed0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0660105660 0x7f06600fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.325 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06601012d0 0x7f06600fe870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 --2- 172.21.15.150:0/3164716077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0660100930 0x7f06600fe330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 >> 172.21.15.150:0/3164716077 conn(0x7f06600774b0 msgr2=0x7f066006a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 shutdown_connections 2024-03-18T16:09:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:32.321+0000 7f06648e8640 1 -- 172.21.15.150:0/3164716077 wait complete. 2024-03-18T16:09:32.337 INFO:tasks.workunit.client.0.smithi150.stdout: 1 963251 10.45 MB/sec execute 377 sec latency 19.618 ms 2024-03-18T16:09:33.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 964814 10.43 MB/sec execute 378 sec latency 88.098 ms 2024-03-18T16:09:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:33 smithi156 ceph-mon[35919]: pgmap v295: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:33 smithi150 ceph-mon[31946]: pgmap v295: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:34.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:33 smithi191 ceph-mon[36784]: pgmap v295: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:34.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 966778 10.44 MB/sec execute 379 sec latency 21.579 ms 2024-03-18T16:09:35.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 968520 10.43 MB/sec execute 380 sec latency 25.395 ms 2024-03-18T16:09:36.014 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:35 smithi191 ceph-mon[36784]: pgmap v296: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:35 smithi156 ceph-mon[35919]: pgmap v296: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:35 smithi150 ceph-mon[31946]: pgmap v296: 97 pgs: 97 active+clean; 562 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-18T16:09:36.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 970525 10.44 MB/sec execute 381 sec latency 13.980 ms 2024-03-18T16:09:37.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 972508 10.43 MB/sec execute 382 sec latency 19.564 ms 2024-03-18T16:09:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:37 smithi156 ceph-mon[35919]: pgmap v297: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 254 op/s 2024-03-18T16:09:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:37 smithi150 ceph-mon[31946]: pgmap v297: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 254 op/s 2024-03-18T16:09:38.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:37 smithi191 ceph-mon[36784]: pgmap v297: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 254 op/s 2024-03-18T16:09:38.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 974062 10.43 MB/sec execute 383 sec latency 92.990 ms 2024-03-18T16:09:39.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:38 smithi150 ceph-mon[31946]: pgmap v298: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 170 op/s 2024-03-18T16:09:39.212 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:38 smithi191 ceph-mon[36784]: pgmap v298: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 170 op/s 2024-03-18T16:09:39.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:38 smithi156 ceph-mon[35919]: pgmap v298: 97 pgs: 97 active+clean; 572 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 170 op/s 2024-03-18T16:09:39.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 975559 10.42 MB/sec execute 384 sec latency 44.212 ms 2024-03-18T16:09:40.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 977494 10.43 MB/sec execute 385 sec latency 15.457 ms 2024-03-18T16:09:41.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 979356 10.42 MB/sec execute 386 sec latency 19.371 ms 2024-03-18T16:09:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:41 smithi191 ceph-mon[36784]: pgmap v299: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 261 op/s 2024-03-18T16:09:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:41 smithi156 ceph-mon[35919]: pgmap v299: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 261 op/s 2024-03-18T16:09:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:41 smithi150 ceph-mon[31946]: pgmap v299: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.3 MiB/s wr, 261 op/s 2024-03-18T16:09:42.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 981110 10.42 MB/sec execute 387 sec latency 10.881 ms 2024-03-18T16:09:43.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 982448 10.41 MB/sec execute 388 sec latency 89.392 ms 2024-03-18T16:09:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:43 smithi191 ceph-mon[36784]: pgmap v300: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:43 smithi156 ceph-mon[35919]: pgmap v300: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:43 smithi150 ceph-mon[31946]: pgmap v300: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:44.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 984548 10.42 MB/sec execute 389 sec latency 22.378 ms 2024-03-18T16:09:45.338 INFO:tasks.workunit.client.0.smithi150.stdout: 1 986312 10.41 MB/sec execute 390 sec latency 19.254 ms 2024-03-18T16:09:45.414 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.412+0000 7fe922485640 1 -- 172.21.15.150:0/87769187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c1017b0 msgr2=0x7fe91c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.412+0000 7fe922485640 1 --2- 172.21.15.150:0/87769187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c1017b0 0x7fe91c101c30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe910009930 tx=0x7fe91002bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- 172.21.15.150:0/87769187 shutdown_connections 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- 172.21.15.150:0/87769187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c102170 0x7fe91c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- 172.21.15.150:0/87769187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c1017b0 0x7fe91c101c30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- 172.21.15.150:0/87769187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 0x7fe91c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- 172.21.15.150:0/87769187 >> 172.21.15.150:0/87769187 conn(0x7fe91c0fd2c0 msgr2=0x7fe91c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- 172.21.15.150:0/87769187 shutdown_connections 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- 172.21.15.150:0/87769187 wait complete. 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 Processor -- start 2024-03-18T16:09:45.415 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- start start 2024-03-18T16:09:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 0x7fe91c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 0x7fe91c19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c111bd0 con 0x7fe91c102170 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.413+0000 7fe922485640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c111a50 con 0x7fe91c1077c0 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe922485640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c111d50 con 0x7fe91c1017b0 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47546/0 (socket says 172.21.15.150:47546) 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 0x7fe91c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 -- 172.21.15.150:0/94284225 learned_addr learned my addr 172.21.15.150:0/94284225 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe920a22640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 0x7fe91c19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 -- 172.21.15.150:0/94284225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 msgr2=0x7fe91c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 0x7fe91c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 -- 172.21.15.150:0/94284225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 msgr2=0x7fe91c19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 0x7fe91c19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 -- 172.21.15.150:0/94284225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c19cff0 con 0x7fe91c102170 2024-03-18T16:09:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91bfff640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 0x7fe91c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe910009230 tx=0x7fe91002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe9197fa640 1 -- 172.21.15.150:0/94284225 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910041070 con 0x7fe91c102170 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91c19d280 con 0x7fe91c102170 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe9197fa640 1 -- 172.21.15.150:0/94284225 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe910004450 con 0x7fe91c102170 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe9197fa640 1 -- 172.21.15.150:0/94284225 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910003de0 con 0x7fe91c102170 2024-03-18T16:09:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.414+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91c1a4b70 con 0x7fe91c102170 2024-03-18T16:09:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe9197fa640 1 -- 172.21.15.150:0/94284225 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fe910044070 con 0x7fe91c102170 2024-03-18T16:09:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 Processor -- start 2024-03-18T16:09:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 -- start start 2024-03-18T16:09:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe9197fa640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 0x7fe9040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da102e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe9197fa640 1 -- 172.21.15.150:0/94284225 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fe9100f0690 con 0x7fe91c102170 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da12c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55dc2d96ba40 con 0x55dc2da07130 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dc2d914920 con 0x55dc2da10820 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe92507c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dc2d967ae0 con 0x55dc2da07a50 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe920a22640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 0x7fe9040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47560/0 (socket says 172.21.15.150:47560) 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe91bfff640 1 -- 172.21.15.150:0/2813915222 learned_addr learned my addr 172.21.15.150:0/2813915222 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.417+0000 7fe920a22640 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da12c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 -- 172.21.15.150:0/2813915222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 msgr2=0x55dc2da102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da102e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.425 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 -- 172.21.15.150:0/2813915222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 msgr2=0x55dc2da12c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da12c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 -- 172.21.15.150:0/2813915222 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2d92c460 con 0x55dc2da07130 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe920a22640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 0x7fe9040a4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe914005fd0 tx=0x7fe914009bf0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0114f0 tx=0x7fe90c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681724a5ee733d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe9027fc640 1 -- 172.21.15.150:0/2813915222 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c01e070 con 0x55dc2da07130 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe9027fc640 1 -- 172.21.15.150:0/2813915222 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe90c011250 con 0x55dc2da07130 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.418+0000 7fe9027fc640 1 -- 172.21.15.150:0/2813915222 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0199b0 con 0x55dc2da07130 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- 172.21.15.150:0/2813915222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 msgr2=0x55dc2da07510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0114f0 tx=0x7fe90c0119b0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- 172.21.15.150:0/2813915222 shutdown_connections 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da12c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 --2- 172.21.15.150:0/2813915222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da07510 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.426 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- 172.21.15.150:0/2813915222 >> 172.21.15.150:0/2813915222 conn(0x55dc2da01a50 msgr2=0x55dc2da01e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- 172.21.15.150:0/2813915222 shutdown_connections 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- 172.21.15.150:0/2813915222 wait complete. 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.419+0000 7fe92507c740 1 -- start start 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 Processor -- start 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da0dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da0e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55dc2da13450 con 0x55dc2da07130 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dc2d927d70 con 0x55dc2da10820 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dc2da135d0 con 0x55dc2da07a50 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe92507c740 10 client.0 _ll_register_callbacks cb 0x55dc2d67fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47572/0 (socket says 172.21.15.150:47572) 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 -- 172.21.15.150:0/1256280727 learned_addr learned my addr 172.21.15.150:0/1256280727 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:45.427 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe920a22640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da0e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da0dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 -- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 msgr2=0x55dc2da0dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da0dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 -- 172.21.15.150:0/1256280727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 msgr2=0x55dc2da0e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da0e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 -- 172.21.15.150:0/1256280727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2d680210 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe91bfff640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c0194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a499d6484e45d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.420+0000 7fe8fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c01e040 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fe90c004050 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe90c011250 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe90c02f050 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 10 client.16212 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dc2d74c620 con 0x55dc2da07130 2024-03-18T16:09:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe90c011250 con 0x55dc2da07130 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fe90c02a030 con 0x55dc2da07130 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 10 client.16212 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 20 client.16212 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da208a0 0x55dc2da22c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 4 client.16212 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc2d978f80 con 0x55dc2da208a0 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.421+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da208a0 0x55dc2da22c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.422+0000 7fe8fb7fe640 10 client.16212 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.422+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da208a0 0x55dc2da22c90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aab5020a7860b2d4 server_cookie=4dedc11d308cffea in_seq=0 out_seq=0 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/1256280727 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc2d978f80 con 0x55dc2da208a0 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe8fb7fe640 10 client.16212 handle_command_reply: tid=0 2024-03-18T16:09:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 client.16212 shutdown 2024-03-18T16:09:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 msgr2=0x55dc2da0d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c0194d0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 shutdown_connections 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da208a0 0x55dc2da22c90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da0e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da0dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe8fb7fe640 0 client.16212 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 --2- 172.21.15.150:0/1256280727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0d820 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 >> 172.21.15.150:0/1256280727 conn(0x55dc2da01a50 msgr2=0x55dc2da037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 shutdown_connections 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 1 -- 172.21.15.150:0/1256280727 wait complete. 2024-03-18T16:09:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.423+0000 7fe92507c740 20 client.16212 trim_cache size 0 max 16384 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 Processor -- start 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 -- start start 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da1c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1afcd0 con 0x55dc2da07130 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1ad5a0 con 0x55dc2da10820 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe92507c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1ab410 con 0x55dc2da07a50 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe91bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47586/0 (socket says 172.21.15.150:47586) 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe91bfff640 1 -- 172.21.15.150:0/134768570 learned_addr learned my addr 172.21.15.150:0/134768570 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da1c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.439+0000 7fe920a22640 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 -- 172.21.15.150:0/134768570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 msgr2=0x55dc2da1c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da1c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 -- 172.21.15.150:0/134768570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 msgr2=0x55dc2da035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 -- 172.21.15.150:0/134768570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2da07880 con 0x55dc2da07130 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe91bfff640 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe90c019540 tx=0x7fe90c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0835a231d1baf8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe901ffb640 1 -- 172.21.15.150:0/134768570 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c01e040 con 0x55dc2da07130 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe901ffb640 1 -- 172.21.15.150:0/134768570 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe90c004b40 con 0x55dc2da07130 2024-03-18T16:09:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.440+0000 7fe901ffb640 1 -- 172.21.15.150:0/134768570 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c019b40 con 0x55dc2da07130 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- 172.21.15.150:0/134768570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 msgr2=0x55dc2da0ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe90c019540 tx=0x7fe90c009f60 comp rx=0 tx=0).stop 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- 172.21.15.150:0/134768570 shutdown_connections 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da10820 0x55dc2da035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da07a50 0x55dc2da1c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 --2- 172.21.15.150:0/134768570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da07130 0x55dc2da0ff50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- 172.21.15.150:0/134768570 >> 172.21.15.150:0/134768570 conn(0x55dc2da01a50 msgr2=0x55dc2da02ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- 172.21.15.150:0/134768570 shutdown_connections 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- 172.21.15.150:0/134768570 wait complete. 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 -- start start 2024-03-18T16:09:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.441+0000 7fe92507c740 1 Processor -- start 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 0x55dc2da14730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 0x55dc2da14c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1aba60 con 0x55dc2da839f0 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1aced0 con 0x55dc2da07a50 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1ac130 con 0x55dc2da10820 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 10 client.0 _ll_register_callbacks cb 0x7fe90c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47590/0 (socket says 172.21.15.150:47590) 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 -- 172.21.15.150:0/2674483105 learned_addr learned my addr 172.21.15.150:0/2674483105 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 0x55dc2da14730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 0x55dc2da14c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 -- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 msgr2=0x55dc2da14c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 0x55dc2da14c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 -- 172.21.15.150:0/2674483105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 msgr2=0x55dc2da14730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 0x55dc2da14730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 -- 172.21.15.150:0/2674483105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2da86320 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 0x55dc2da14730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe91b7fe640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 0x55dc2da14c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe920a22640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe91400fcc0 tx=0x7fe914004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c84d201f6074f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe8fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91402f060 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fe9140046c0 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.442+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe90c004b40 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe914010430 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 10 client.16218 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dc2da87560 con 0x55dc2da839f0 2024-03-18T16:09:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe90c004b40 con 0x55dc2da839f0 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fe914029070 con 0x55dc2da839f0 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 10 client.16218 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 20 client.16218 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da87d10 0x55dc2da8a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 4 client.16218 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dc2d9732d0 con 0x55dc2da87d10 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.443+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da87d10 0x55dc2da8a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.444+0000 7fe8fb7fe640 10 client.16218 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.444+0000 7fe91bfff640 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da87d10 0x55dc2da8a130 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd80ae3e205f2924 server_cookie=b454ab3ef41d8360 in_seq=0 out_seq=0 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.454+0000 7fe8fb7fe640 1 -- 172.21.15.150:0/2674483105 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dc2d9732d0 con 0x55dc2da87d10 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.454+0000 7fe8fb7fe640 10 client.16218 handle_command_reply: tid=0 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.454+0000 7fe92507c740 1 client.16218 shutdown 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 msgr2=0x55dc2da85de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe91400fcc0 tx=0x7fe914004ab0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 shutdown_connections 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dc2da87d10 0x55dc2da8a130 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe8fb7fe640 0 client.16218 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc2da839f0 0x55dc2da85de0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dc2da10820 0x55dc2da14c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 --2- 172.21.15.150:0/2674483105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dc2da07a50 0x55dc2da14730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 >> 172.21.15.150:0/2674483105 conn(0x55dc2da01a50 msgr2=0x55dc2da00320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 shutdown_connections 2024-03-18T16:09:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 1 -- 172.21.15.150:0/2674483105 wait complete. 2024-03-18T16:09:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.455+0000 7fe92507c740 20 client.16218 trim_cache size 0 max 16384 2024-03-18T16:09:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:09:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:09:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:09:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 msgr2=0x7fe9040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 0x7fe9040a4ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe914005fd0 tx=0x7fe914009bf0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 msgr2=0x7fe91c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe910009230 tx=0x7fe91002cb30 comp rx=0 tx=0).stop 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 shutdown_connections 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe9040a2a10 0x7fe9040a4ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe91c1077c0 0x7fe91c19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe91c102170 0x7fe91c198590 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 --2- 172.21.15.150:0/94284225 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe91c1017b0 0x7fe91c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 >> 172.21.15.150:0/94284225 conn(0x7fe91c0fd2c0 msgr2=0x7fe91c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 shutdown_connections 2024-03-18T16:09:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:45.456+0000 7fe922485640 1 -- 172.21.15.150:0/94284225 wait complete. 2024-03-18T16:09:45.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:45 smithi191 ceph-mon[36784]: pgmap v301: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:46.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:45 smithi156 ceph-mon[35919]: pgmap v301: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:45 smithi150 ceph-mon[31946]: pgmap v301: 97 pgs: 97 active+clean; 602 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 174 op/s 2024-03-18T16:09:46.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 988233 10.41 MB/sec execute 391 sec latency 10.498 ms 2024-03-18T16:09:47.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 990020 10.41 MB/sec execute 392 sec latency 24.402 ms 2024-03-18T16:09:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:47 smithi191 ceph-mon[36784]: pgmap v302: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 266 op/s 2024-03-18T16:09:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:47 smithi156 ceph-mon[35919]: pgmap v302: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 266 op/s 2024-03-18T16:09:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:47 smithi150 ceph-mon[31946]: pgmap v302: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 266 op/s 2024-03-18T16:09:48.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 991700 10.41 MB/sec execute 393 sec latency 30.813 ms 2024-03-18T16:09:49.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 992923 10.40 MB/sec execute 394 sec latency 89.555 ms 2024-03-18T16:09:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:49 smithi191 ceph-mon[36784]: pgmap v303: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 182 op/s 2024-03-18T16:09:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:49 smithi156 ceph-mon[35919]: pgmap v303: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 182 op/s 2024-03-18T16:09:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:49 smithi150 ceph-mon[31946]: pgmap v303: 97 pgs: 97 active+clean; 611 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.8 MiB/s wr, 182 op/s 2024-03-18T16:09:50.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 995042 10.40 MB/sec execute 395 sec latency 20.018 ms 2024-03-18T16:09:51.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 996478 10.39 MB/sec execute 396 sec latency 22.509 ms 2024-03-18T16:09:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:51 smithi191 ceph-mon[36784]: pgmap v304: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.4 MiB/s wr, 273 op/s 2024-03-18T16:09:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:51 smithi156 ceph-mon[35919]: pgmap v304: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.4 MiB/s wr, 273 op/s 2024-03-18T16:09:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:51 smithi150 ceph-mon[31946]: pgmap v304: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 6.4 MiB/s wr, 273 op/s 2024-03-18T16:09:52.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 998714 10.40 MB/sec execute 397 sec latency 17.175 ms 2024-03-18T16:09:53.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1000197 10.39 MB/sec execute 398 sec latency 21.886 ms 2024-03-18T16:09:53.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:53 smithi191 ceph-mon[36784]: pgmap v305: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:53 smithi156 ceph-mon[35919]: pgmap v305: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:53 smithi150 ceph-mon[31946]: pgmap v305: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:54.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1002153 10.40 MB/sec execute 399 sec latency 103.082 ms 2024-03-18T16:09:55.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1003568 10.38 MB/sec execute 400 sec latency 20.140 ms 2024-03-18T16:09:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:55 smithi191 ceph-mon[36784]: pgmap v306: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:55 smithi156 ceph-mon[35919]: pgmap v306: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:55 smithi150 ceph-mon[31946]: pgmap v306: 97 pgs: 97 active+clean; 620 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 182 op/s 2024-03-18T16:09:56.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1005732 10.39 MB/sec execute 401 sec latency 17.498 ms 2024-03-18T16:09:57.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1007109 10.38 MB/sec execute 402 sec latency 24.228 ms 2024-03-18T16:09:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:57 smithi191 ceph-mon[36784]: pgmap v307: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:09:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:57 smithi156 ceph-mon[35919]: pgmap v307: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:09:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:57 smithi150 ceph-mon[31946]: pgmap v307: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:09:58.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1009358 10.39 MB/sec execute 403 sec latency 16.089 ms 2024-03-18T16:09:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- 172.21.15.150:0/2269028548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd48809ad00 msgr2=0x7fd48809b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- 172.21.15.150:0/2269028548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd48809ad00 0x7fd48809b0e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009900 tx=0x7fd48c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- 172.21.15.150:0/2269028548 shutdown_connections 2024-03-18T16:09:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- 172.21.15.150:0/2269028548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd488006f60 0x7fd4880a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- 172.21.15.150:0/2269028548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 0x7fd488006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- 172.21.15.150:0/2269028548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd48809ad00 0x7fd48809b0e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- 172.21.15.150:0/2269028548 >> 172.21.15.150:0/2269028548 conn(0x7fd48800a7e0 msgr2=0x7fd48800abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- 172.21.15.150:0/2269028548 shutdown_connections 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- 172.21.15.150:0/2269028548 wait complete. 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 Processor -- start 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- start start 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 0x7fd488094f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48809ad00 0x7fd4880959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880a60f0 con 0x7fd488006f60 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880a5f70 con 0x7fd4880065a0 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.560+0000 7fd4961c5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880a6270 con 0x7fd48809ad00 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.565 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40182/0 (socket says 172.21.15.150:40182) 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4951c3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 0x7fd488094f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 -- 172.21.15.150:0/1127891055 learned_addr learned my addr 172.21.15.150:0/1127891055 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4959c4640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48809ad00 0x7fd4880959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 -- 172.21.15.150:0/1127891055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48809ad00 msgr2=0x7fd4880959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48809ad00 0x7fd4880959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 -- 172.21.15.150:0/1127891055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 msgr2=0x7fd488094f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 0x7fd488094f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 -- 172.21.15.150:0/1127891055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48813aa30 con 0x7fd488006f60 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd48000e9f0 tx=0x7fd48000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4867fc640 1 -- 172.21.15.150:0/1127891055 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48000cd80 con 0x7fd488006f60 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4867fc640 1 -- 172.21.15.150:0/1127891055 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd480010910 con 0x7fd488006f60 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48813ad20 con 0x7fd488006f60 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4867fc640 1 -- 172.21.15.150:0/1127891055 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480018990 con 0x7fd488006f60 2024-03-18T16:09:58.566 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.561+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48813b260 con 0x7fd488006f60 2024-03-18T16:09:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.563+0000 7fd4867fc640 1 -- 172.21.15.150:0/1127891055 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fd480010ac0 con 0x7fd488006f60 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 Processor -- start 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 -- start start 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4867fc640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 0x7fd47c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4959c4640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 0x7fd47c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98ec4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560db9847a40 con 0x560db98e32f0 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4867fc640 1 -- 172.21.15.150:0/1127891055 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fd4800cf100 con 0x7fd488006f60 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560db97f0920 con 0x560db98ec9e0 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd49a015740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560db9843ae0 con 0x560db98e3c10 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4951c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.573 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4951c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40190/0 (socket says 172.21.15.150:40190) 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4951c3640 1 -- 172.21.15.150:0/1503552926 learned_addr learned my addr 172.21.15.150:0/1503552926 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98ec4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.564+0000 7fd4959c4640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 -- 172.21.15.150:0/1503552926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 msgr2=0x560db98ec4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98ec4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 -- 172.21.15.150:0/1503552926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 msgr2=0x560db98eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 -- 172.21.15.150:0/1503552926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560db9808460 con 0x560db98e32f0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98ec4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4959c4640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 0x7fd47c0a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd490067090 tx=0x7fd49007f810 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009900 tx=0x7fd48c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436549b9d00287d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd47b7fe640 1 -- 172.21.15.150:0/1503552926 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c045070 con 0x560db98e32f0 2024-03-18T16:09:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd47b7fe640 1 -- 172.21.15.150:0/1503552926 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd48c02c880 con 0x560db98e32f0 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.565+0000 7fd47b7fe640 1 -- 172.21.15.150:0/1503552926 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c004bf0 con 0x560db98e32f0 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- 172.21.15.150:0/1503552926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 msgr2=0x560db98e36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009900 tx=0x7fd48c0095e0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- 172.21.15.150:0/1503552926 shutdown_connections 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98ec4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 --2- 172.21.15.150:0/1503552926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e32f0 0x560db98e36d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- 172.21.15.150:0/1503552926 >> 172.21.15.150:0/1503552926 conn(0x560db98ddc10 msgr2=0x560db98ddff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- 172.21.15.150:0/1503552926 shutdown_connections 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- 172.21.15.150:0/1503552926 wait complete. 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 -- start start 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.566+0000 7fd49a015740 1 Processor -- start 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e32f0 0x560db98e9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x560db98ef5b0 con 0x560db98e3c10 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560db9803d70 con 0x560db98ec9e0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x560db98ef730 con 0x560db98e32f0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 10 client.0 _ll_register_callbacks cb 0x560db955ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40202/0 (socket says 172.21.15.150:40202) 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4951c3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e32f0 0x560db98e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 -- 172.21.15.150:0/3861771953 learned_addr learned my addr 172.21.15.150:0/3861771953 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4959c4640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 -- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e32f0 msgr2=0x560db98e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e32f0 0x560db98e9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 -- 172.21.15.150:0/3861771953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 msgr2=0x560db98ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 -- 172.21.15.150:0/3861771953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560db955c210 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd4949c2640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd4800cc900 tx=0x7fd48009b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163395e8d09fe8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4800b4060 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fd4800afb40 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd48c02c880 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd4800afd40 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 10 client.16230 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.567+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560db9628620 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd48c02c880 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd48009acf0 con 0x560db98e3c10 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 10 client.16230 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 10 client.16230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 20 client.16230 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db98fcb60 0x560db98fef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 4 client.16230 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560db9854f80 con 0x560db98fcb60 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd4951c3640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db98fcb60 0x560db98fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd478ff9640 10 client.16230 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.568+0000 7fd4951c3640 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db98fcb60 0x560db98fef50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3120986fe1aa17d server_cookie=93fd74c9fa0983bc in_seq=0 out_seq=0 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd478ff9640 1 -- 172.21.15.150:0/3861771953 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560db9854f80 con 0x560db98fcb60 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd478ff9640 10 client.16230 handle_command_reply: tid=0 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 client.16230 shutdown 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 msgr2=0x560db98e9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd4800cc900 tx=0x7fd48009b950 comp rx=0 tx=0).stop 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 shutdown_connections 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db98fcb60 0x560db98fef50 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98ec9e0 0x560db98ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e9ed0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 --2- 172.21.15.150:0/3861771953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e32f0 0x560db98e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd478ff9640 0 client.16230 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 >> 172.21.15.150:0/3861771953 conn(0x560db98ddc10 msgr2=0x560db98df960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 shutdown_connections 2024-03-18T16:09:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 1 -- 172.21.15.150:0/3861771953 wait complete. 2024-03-18T16:09:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.569+0000 7fd49a015740 20 client.16230 trim_cache size 0 max 16384 2024-03-18T16:09:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.584+0000 7fd49a015740 1 Processor -- start 2024-03-18T16:09:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.584+0000 7fd49a015740 1 -- start start 2024-03-18T16:09:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98ec050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98f8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881463f0 con 0x560db98ec9e0 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd488143cc0 con 0x560db98e32f0 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd49a015740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd488141b30 con 0x560db98e3c10 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40216/0 (socket says 172.21.15.150:40216) 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 -- 172.21.15.150:0/3748357841 learned_addr learned my addr 172.21.15.150:0/3748357841 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4949c2640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4951c3640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98ec050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 -- 172.21.15.150:0/3748357841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 msgr2=0x560db98f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98f8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 -- 172.21.15.150:0/3748357841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 msgr2=0x560db98ec050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98ec050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 -- 172.21.15.150:0/3748357841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560db98e7ee0 con 0x560db98ec9e0 2024-03-18T16:09:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4951c3640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98ec050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.585+0000 7fd4959c4640 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd4900674c0 tx=0x7fd49007a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9419580fc7660815 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.586+0000 7fd47affd640 1 -- 172.21.15.150:0/3748357841 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490088040 con 0x560db98ec9e0 2024-03-18T16:09:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.586+0000 7fd47affd640 1 -- 172.21.15.150:0/3748357841 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd490069040 con 0x560db98ec9e0 2024-03-18T16:09:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.586+0000 7fd47affd640 1 -- 172.21.15.150:0/3748357841 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49008c580 con 0x560db98ec9e0 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- 172.21.15.150:0/3748357841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 msgr2=0x560db9904860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd4900674c0 tx=0x7fd49007a000 comp rx=0 tx=0).stop 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- 172.21.15.150:0/3748357841 shutdown_connections 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98ec9e0 0x560db9904860 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98e3c10 0x560db98f8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 --2- 172.21.15.150:0/3748357841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98ec050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- 172.21.15.150:0/3748357841 >> 172.21.15.150:0/3748357841 conn(0x560db98ddc10 msgr2=0x560db98e9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- 172.21.15.150:0/3748357841 shutdown_connections 2024-03-18T16:09:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- 172.21.15.150:0/3748357841 wait complete. 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.587+0000 7fd49a015740 1 -- start start 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 Processor -- start 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98ec9e0 0x560db996a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd488142180 con 0x560db98e3c10 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881435f0 con 0x560db98e32f0 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fd488142850 con 0x560db98ec9e0 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd49a015740 10 client.0 _ll_register_callbacks cb 0x7fd48000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd4949c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40224/0 (socket says 172.21.15.150:40224) 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd4949c2640 1 -- 172.21.15.150:0/1850802081 learned_addr learned my addr 172.21.15.150:0/1850802081 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:09:58.594 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.588+0000 7fd4959c4640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98ec9e0 0x560db996a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 -- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98ec9e0 msgr2=0x560db996a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98ec9e0 0x560db996a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 -- 172.21.15.150:0/1850802081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 msgr2=0x560db98e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98e6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 -- 172.21.15.150:0/1850802081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560db996abe0 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd4949c2640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd48009b510 tx=0x7fd4800cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bf716ff6668517 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48009b170 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8527+0+0 (secure 0 0 0) 0x7fd4800afb40 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd490069040 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd480018b30 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd49a015740 10 client.16236 fetch_fsmap learned FSMap version 31 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.589+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560db996bdf0 con 0x560db98e3c10 2024-03-18T16:09:58.595 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd490069040 con 0x560db98e3c10 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd48009a890 con 0x560db98e3c10 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 10 client.16236 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 20 client.16236 populate_metadata read hostname 'smithi150' 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db996c4b0 0x560db996e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 4 client.16236 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560db984f2d0 con 0x560db996c4b0 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db996c4b0 0x560db996e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd4951c3640 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db996c4b0 0x560db996e8d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=218f4812054cb0ed server_cookie=7b0b07fd3648ce3c in_seq=0 out_seq=0 2024-03-18T16:09:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.590+0000 7fd478ff9640 10 client.16236 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.602+0000 7fd478ff9640 1 -- 172.21.15.150:0/1850802081 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560db984f2d0 con 0x560db996c4b0 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.602+0000 7fd478ff9640 10 client.16236 handle_command_reply: tid=0 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.602+0000 7fd49a015740 1 client.16236 shutdown 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 msgr2=0x560db98e7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd48009b510 tx=0x7fd4800cf040 comp rx=0 tx=0).stop 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 shutdown_connections 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x560db996c4b0 0x560db996e8d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x560db98ec9e0 0x560db996a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560db98e3c10 0x560db98e7080 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd478ff9640 0 client.16236 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 --2- 172.21.15.150:0/1850802081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560db98e32f0 0x560db98e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 >> 172.21.15.150:0/1850802081 conn(0x560db98ddc10 msgr2=0x560db98ea1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 shutdown_connections 2024-03-18T16:09:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 1 -- 172.21.15.150:0/1850802081 wait complete. 2024-03-18T16:09:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd49a015740 20 client.16236 trim_cache size 0 max 16384 2024-03-18T16:09:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:09:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:09:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:09:58.607 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 msgr2=0x7fd47c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 0x7fd47c0a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd490067090 tx=0x7fd49007f810 comp rx=0 tx=0).stop 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 msgr2=0x7fd488095460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd48000e9f0 tx=0x7fd48000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 shutdown_connections 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fd47c0a29f0 0x7fd47c0a4eb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd48809ad00 0x7fd4880959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd488006f60 0x7fd488095460 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 --2- 172.21.15.150:0/1127891055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4880065a0 0x7fd488094f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.603+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 >> 172.21.15.150:0/1127891055 conn(0x7fd48800a7e0 msgr2=0x7fd488092e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.604+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 shutdown_connections 2024-03-18T16:09:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:09:58.604+0000 7fd4961c5640 1 -- 172.21.15.150:0/1127891055 wait complete. 2024-03-18T16:09:59.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1010616 10.38 MB/sec execute 404 sec latency 84.807 ms 2024-03-18T16:09:59.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:09:59 smithi191 ceph-mon[36784]: pgmap v308: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:00.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:09:59 smithi156 ceph-mon[35919]: pgmap v308: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:09:59 smithi150 ceph-mon[31946]: pgmap v308: 97 pgs: 97 active+clean; 629 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:00.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1012790 10.38 MB/sec execute 405 sec latency 18.577 ms 2024-03-18T16:10:00.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:00 smithi191 ceph-mon[36784]: overall HEALTH_OK 2024-03-18T16:10:01.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:00 smithi156 ceph-mon[35919]: overall HEALTH_OK 2024-03-18T16:10:01.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:00 smithi150 ceph-mon[31946]: overall HEALTH_OK 2024-03-18T16:10:01.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1013725 10.36 MB/sec execute 406 sec latency 23.612 ms 2024-03-18T16:10:01.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:01 smithi191 ceph-mon[36784]: pgmap v309: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:10:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:01 smithi156 ceph-mon[35919]: pgmap v309: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:10:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:01 smithi150 ceph-mon[31946]: pgmap v309: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 269 op/s 2024-03-18T16:10:02.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1015843 10.37 MB/sec execute 407 sec latency 20.643 ms 2024-03-18T16:10:03.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1017449 10.37 MB/sec execute 408 sec latency 22.308 ms 2024-03-18T16:10:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:03 smithi191 ceph-mon[36784]: pgmap v310: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:10:03.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:03 smithi191 ceph-mon[36784]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T16:10:04.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:03 smithi156 ceph-mon[35919]: pgmap v310: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:10:04.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:03 smithi156 ceph-mon[35919]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T16:10:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:03 smithi150 ceph-mon[31946]: pgmap v310: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:10:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:03 smithi150 ceph-mon[31946]: osdmap e98: 12 total, 12 up, 12 in 2024-03-18T16:10:04.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1018816 10.36 MB/sec execute 409 sec latency 122.329 ms 2024-03-18T16:10:05.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1020510 10.35 MB/sec execute 410 sec latency 19.081 ms 2024-03-18T16:10:05.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:05 smithi150 ceph-mon[31946]: pgmap v312: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 214 op/s 2024-03-18T16:10:05.712 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:05 smithi191 ceph-mon[36784]: pgmap v312: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 214 op/s 2024-03-18T16:10:05.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:05 smithi156 ceph-mon[35919]: pgmap v312: 97 pgs: 97 active+clean; 595 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 214 op/s 2024-03-18T16:10:06.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1022579 10.36 MB/sec execute 411 sec latency 21.784 ms 2024-03-18T16:10:07.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1024002 10.35 MB/sec execute 412 sec latency 17.708 ms 2024-03-18T16:10:07.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:07 smithi191 ceph-mon[36784]: pgmap v313: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:07 smithi156 ceph-mon[35919]: pgmap v313: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:07 smithi150 ceph-mon[31946]: pgmap v313: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:08.339 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1026060 10.35 MB/sec execute 413 sec latency 17.984 ms 2024-03-18T16:10:09.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1027355 10.34 MB/sec execute 414 sec latency 45.665 ms 2024-03-18T16:10:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:09 smithi191 ceph-mon[36784]: pgmap v314: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:09.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:09 smithi191 ceph-mon[36784]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T16:10:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:09 smithi156 ceph-mon[35919]: pgmap v314: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:09 smithi156 ceph-mon[35919]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T16:10:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:09 smithi150 ceph-mon[31946]: pgmap v314: 97 pgs: 97 active+clean; 578 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 260 op/s 2024-03-18T16:10:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:09 smithi150 ceph-mon[31946]: osdmap e99: 12 total, 12 up, 12 in 2024-03-18T16:10:10.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1029112 10.34 MB/sec execute 415 sec latency 20.886 ms 2024-03-18T16:10:11.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1030819 10.34 MB/sec execute 416 sec latency 11.079 ms 2024-03-18T16:10:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 -- 172.21.15.150:0/1636869953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f80100930 msgr2=0x7f8f80100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/1636869953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f80100930 0x7f8f80100db0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009900 tx=0x7f8f6802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 -- 172.21.15.150:0/1636869953 shutdown_connections 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/1636869953 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f801012f0 0x7f8f8010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/1636869953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f80100930 0x7f8f80100db0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/1636869953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f80105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.694+0000 7f8f84c7e640 1 -- 172.21.15.150:0/1636869953 >> 172.21.15.150:0/1636869953 conn(0x7f8f80077530 msgr2=0x7f8f80077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.695+0000 7f8f84c7e640 1 -- 172.21.15.150:0/1636869953 shutdown_connections 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.695+0000 7f8f84c7e640 1 -- 172.21.15.150:0/1636869953 wait complete. 2024-03-18T16:10:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.695+0000 7f8f84c7e640 1 Processor -- start 2024-03-18T16:10:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.695+0000 7f8f84c7e640 1 -- start start 2024-03-18T16:10:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f80100930 0x7f8f80198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f801a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80110b60 con 0x7f8f801012f0 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801109e0 con 0x7f8f80105680 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f84c7e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8f80110ce0 con 0x7f8f80100930 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f80100930 0x7f8f80198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33582/0 (socket says 172.21.15.150:33582) 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 -- 172.21.15.150:0/3542187063 learned_addr learned my addr 172.21.15.150:0/3542187063 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f801a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 -- 172.21.15.150:0/3542187063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f80100930 msgr2=0x7f8f80198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f80100930 0x7f8f80198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 -- 172.21.15.150:0/3542187063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 msgr2=0x7f8f801a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f801a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 -- 172.21.15.150:0/3542187063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f801a44a0 con 0x7f8f801012f0 2024-03-18T16:10:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f801a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8f68005940 tx=0x7f8f68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.696+0000 7f8f77fff640 1 -- 172.21.15.150:0/3542187063 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68041070 con 0x7f8f801012f0 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.697+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f801a4670 con 0x7f8f801012f0 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.697+0000 7f8f77fff640 1 -- 172.21.15.150:0/3542187063 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f680054e0 con 0x7f8f801012f0 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.697+0000 7f8f77fff640 1 -- 172.21.15.150:0/3542187063 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68003d00 con 0x7f8f801012f0 2024-03-18T16:10:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.697+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f801a4ad0 con 0x7f8f801012f0 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f77fff640 1 -- 172.21.15.150:0/3542187063 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f8f68044070 con 0x7f8f801012f0 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 Processor -- start 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 -- start start 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 0x561b19db0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19db92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561b19d14a40 con 0x561b19db9820 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561b19cbd920 con 0x561b19db0130 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f87867740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561b19d10ae0 con 0x561b19db0a50 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.699+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33596/0 (socket says 172.21.15.150:33596) 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 0x561b19db0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19db92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f75d9b640 1 -- 172.21.15.150:0/11525896 learned_addr learned my addr 172.21.15.150:0/11525896 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f77fff640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 0x7f8f6c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 -- 172.21.15.150:0/11525896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 msgr2=0x561b19db92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19db92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 -- 172.21.15.150:0/11525896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 msgr2=0x561b19dbbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 -- 172.21.15.150:0/11525896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b19cd5460 con 0x561b19db0130 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f77fff640 1 -- 172.21.15.150:0/3542187063 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f8f680f07e0 con 0x7f8f801012f0 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 0x7f8f6c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 0x561b19db0510 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f78011850 tx=0x7f8f78011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dbd0c80d60d070a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f74d99640 1 -- 172.21.15.150:0/11525896 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78017050 con 0x561b19db0130 2024-03-18T16:10:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f74d99640 1 -- 172.21.15.150:0/11525896 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f78026440 con 0x561b19db0130 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 0x7f8f6c0a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f78023000 tx=0x7f8f78024000 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.700+0000 7f8f74d99640 1 -- 172.21.15.150:0/11525896 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800e610 con 0x561b19db0130 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 -- 172.21.15.150:0/11525896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 msgr2=0x561b19db0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 0x561b19db0510 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f78011850 tx=0x7f8f78011d10 comp rx=0 tx=0).stop 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 -- 172.21.15.150:0/11525896 shutdown_connections 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db9820 0x561b19dbbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19db92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 --2- 172.21.15.150:0/11525896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0130 0x561b19db0510 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 -- 172.21.15.150:0/11525896 >> 172.21.15.150:0/11525896 conn(0x561b19daaa50 msgr2=0x561b19daae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 -- 172.21.15.150:0/11525896 shutdown_connections 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.701+0000 7f8f87867740 1 -- 172.21.15.150:0/11525896 wait complete. 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 -- start start 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 Processor -- start 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0a50 0x561b19db6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 0x561b19db7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561b19dc0860 con 0x561b19db0130 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561b19cd0d70 con 0x561b19db0a50 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x561b19dc09e0 con 0x561b19db9820 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f87867740 10 client.0 _ll_register_callbacks cb 0x561b19a28a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33612/0 (socket says 172.21.15.150:33612) 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 -- 172.21.15.150:0/1813863232 learned_addr learned my addr 172.21.15.150:0/1813863232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 0x561b19db7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0a50 0x561b19db6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 -- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 msgr2=0x561b19db7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 0x561b19db7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 -- 172.21.15.150:0/1813863232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0a50 msgr2=0x561b19db6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0a50 0x561b19db6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 -- 172.21.15.150:0/1813863232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b19a29210 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 0x561b19db7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f700126e0 tx=0x7f8f70012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c9c08df5dbaf97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.702+0000 7f8f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7000f670 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f8f7000f810 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f78026440 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f7000fa10 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 10 client.16248 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b19af5620 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f78026440 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f8f7001c420 con 0x561b19db0130 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 10 client.16248 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 10 client.16248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 20 client.16248 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19dcdef0 0x561b19dd02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 4 client.16248 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b19d21f80 con 0x561b19dcdef0 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.703+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19dcdef0 0x561b19dd02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.704+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19dcdef0 0x561b19dd02e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f188127572cb405a server_cookie=4ab29deafe723a26 in_seq=0 out_seq=0 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.704+0000 7f8f65ffb640 10 client.16248 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f65ffb640 1 -- 172.21.15.150:0/1813863232 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b19d21f80 con 0x561b19dcdef0 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f65ffb640 10 client.16248 handle_command_reply: tid=0 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 client.16248 shutdown 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 msgr2=0x561b19db68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f700126e0 tx=0x7f8f70012ae0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 shutdown_connections 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19dcdef0 0x561b19dd02e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db9820 0x561b19db7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db0a50 0x561b19db6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 --2- 172.21.15.150:0/1813863232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19db68f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f65ffb640 0 client.16248 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 >> 172.21.15.150:0/1813863232 conn(0x561b19daaa50 msgr2=0x561b19dac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 shutdown_connections 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 1 -- 172.21.15.150:0/1813863232 wait complete. 2024-03-18T16:10:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.705+0000 7f8f87867740 20 client.16248 trim_cache size 0 max 16384 2024-03-18T16:10:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 Processor -- start 2024-03-18T16:10:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 -- start start 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0130 0x561b19db9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19da5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801adb00 con 0x561b19db0a50 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801ab3d0 con 0x561b19db9820 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f87867740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9240 con 0x561b19db0130 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33620/0 (socket says 172.21.15.150:33620) 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 -- 172.21.15.150:0/2060607533 learned_addr learned my addr 172.21.15.150:0/2060607533 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0130 0x561b19db9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19da5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 -- 172.21.15.150:0/2060607533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0130 msgr2=0x561b19db9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0130 0x561b19db9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 -- 172.21.15.150:0/2060607533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 msgr2=0x561b19da5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19da5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.720+0000 7f8f75d9b640 1 -- 172.21.15.150:0/2060607533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b19db58a0 con 0x561b19db0a50 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8f680ee0f0 tx=0x7f8f680f44c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba745beb70c5b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f67fff640 1 -- 172.21.15.150:0/2060607533 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6804b040 con 0x561b19db0a50 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f67fff640 1 -- 172.21.15.150:0/2060607533 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6802ccd0 con 0x561b19db0a50 2024-03-18T16:10:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f67fff640 1 -- 172.21.15.150:0/2060607533 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f680bd000 con 0x561b19db0a50 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 -- 172.21.15.150:0/2060607533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 msgr2=0x561b19dc9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8f680ee0f0 tx=0x7f8f680f44c0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 -- 172.21.15.150:0/2060607533 shutdown_connections 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19da5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0a50 0x561b19dc9600 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 --2- 172.21.15.150:0/2060607533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0130 0x561b19db9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 -- 172.21.15.150:0/2060607533 >> 172.21.15.150:0/2060607533 conn(0x561b19daaa50 msgr2=0x561b19da9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.727 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.721+0000 7f8f87867740 1 -- 172.21.15.150:0/2060607533 shutdown_connections 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 -- 172.21.15.150:0/2060607533 wait complete. 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 -- start start 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 Processor -- start 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19e2da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19dc1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9890 con 0x561b19db0130 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801aad00 con 0x561b19db9820 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8f801a9f60 con 0x561b19db0a50 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f87867740 10 client.0 _ll_register_callbacks cb 0x7f8f70009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33632/0 (socket says 172.21.15.150:33632) 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 -- 172.21.15.150:0/765246420 learned_addr learned my addr 172.21.15.150:0/765246420 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19e2da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.728 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7ed9d640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19dc1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 -- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 msgr2=0x561b19e2da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19e2da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 -- 172.21.15.150:0/765246420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 msgr2=0x561b19dc1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19dc1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.722+0000 7f8f7e59c640 1 -- 172.21.15.150:0/765246420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b19db0880 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f7e59c640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f7001c8e0 tx=0x7f8f7001c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c293020379e73f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7000fdc0 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f8f7001cad0 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b19d1c2d0 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f7001ccd0 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 10 client.16254 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b19e34b10 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561b19d1c2d0 con 0x561b19db0130 2024-03-18T16:10:11.729 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f8f700038f0 con 0x561b19db0130 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 10 client.16254 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 20 client.16254 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19e352c0 0x561b19e376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 4 client.16254 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b19c9a770 con 0x561b19e352c0 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.723+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19e352c0 0x561b19e376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.724+0000 7f8f65ffb640 10 client.16254 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.724+0000 7f8f75d9b640 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19e352c0 0x561b19e376e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=622d33df829e0a0e server_cookie=d05f408cf99435fc in_seq=0 out_seq=0 2024-03-18T16:10:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.735+0000 7f8f65ffb640 1 -- 172.21.15.150:0/765246420 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b19c9a770 con 0x561b19e352c0 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.735+0000 7f8f65ffb640 10 client.16254 handle_command_reply: tid=0 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.735+0000 7f8f87867740 1 client.16254 shutdown 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.735+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 msgr2=0x561b19e2d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.735+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f7001c8e0 tx=0x7f8f7001c650 comp rx=0 tx=0).stop 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 shutdown_connections 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x561b19e352c0 0x561b19e376e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b19db9820 0x561b19dc1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x561b19db0a50 0x561b19e2da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f65ffb640 0 client.16254 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 --2- 172.21.15.150:0/765246420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b19db0130 0x561b19e2d530 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 >> 172.21.15.150:0/765246420 conn(0x561b19daaa50 msgr2=0x561b19db6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 shutdown_connections 2024-03-18T16:10:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 1 -- 172.21.15.150:0/765246420 wait complete. 2024-03-18T16:10:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f87867740 20 client.16254 trim_cache size 0 max 16384 2024-03-18T16:10:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:10:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:10:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:10:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 msgr2=0x7f8f6c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 0x7f8f6c0a4ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f78023000 tx=0x7f8f78024000 comp rx=0 tx=0).stop 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 msgr2=0x7f8f80199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8f68005940 tx=0x7f8f68004270 comp rx=0 tx=0).stop 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 shutdown_connections 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f8f6c0a2a10 0x7f8f6c0a4ed0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f80105680 0x7f8f801a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f801012f0 0x7f8f80199420 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 --2- 172.21.15.150:0/3542187063 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f80100930 0x7f8f80198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 >> 172.21.15.150:0/3542187063 conn(0x7f8f80077530 msgr2=0x7f8f800fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 shutdown_connections 2024-03-18T16:10:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:11.736+0000 7f8f84c7e640 1 -- 172.21.15.150:0/3542187063 wait complete. 2024-03-18T16:10:11.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:11 smithi191 ceph-mon[36784]: pgmap v316: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.1 MiB/s wr, 318 op/s 2024-03-18T16:10:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:11 smithi156 ceph-mon[35919]: pgmap v316: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.1 MiB/s wr, 318 op/s 2024-03-18T16:10:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:11 smithi150 ceph-mon[31946]: pgmap v316: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.1 MiB/s wr, 318 op/s 2024-03-18T16:10:12.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1032416 10.33 MB/sec execute 417 sec latency 25.374 ms 2024-03-18T16:10:13.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1034253 10.34 MB/sec execute 418 sec latency 15.775 ms 2024-03-18T16:10:13.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:13 smithi191 ceph-mon[36784]: pgmap v317: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 256 op/s 2024-03-18T16:10:14.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:13 smithi156 ceph-mon[35919]: pgmap v317: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 256 op/s 2024-03-18T16:10:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:13 smithi150 ceph-mon[31946]: pgmap v317: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 256 op/s 2024-03-18T16:10:14.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1035639 10.32 MB/sec execute 419 sec latency 116.986 ms 2024-03-18T16:10:15.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1037717 10.33 MB/sec execute 420 sec latency 19.727 ms 2024-03-18T16:10:15.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:15 smithi191 ceph-mon[36784]: pgmap v318: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-18T16:10:16.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:15 smithi156 ceph-mon[35919]: pgmap v318: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-18T16:10:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:15 smithi150 ceph-mon[31946]: pgmap v318: 97 pgs: 97 active+clean; 588 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-18T16:10:16.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1039100 10.32 MB/sec execute 421 sec latency 35.258 ms 2024-03-18T16:10:17.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1041261 10.33 MB/sec execute 422 sec latency 18.428 ms 2024-03-18T16:10:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:17 smithi191 ceph-mon[36784]: pgmap v319: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:17 smithi156 ceph-mon[35919]: pgmap v319: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:17 smithi150 ceph-mon[31946]: pgmap v319: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:18.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1043016 10.32 MB/sec execute 423 sec latency 17.631 ms 2024-03-18T16:10:19.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1044791 10.33 MB/sec execute 424 sec latency 72.428 ms 2024-03-18T16:10:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:19 smithi191 ceph-mon[36784]: pgmap v320: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:19 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:19 smithi156 ceph-mon[35919]: pgmap v320: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:19 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:20.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:19 smithi150 ceph-mon[31946]: pgmap v320: 97 pgs: 97 active+clean; 602 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 215 op/s 2024-03-18T16:10:20.089 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:19 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:20.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1046107 10.31 MB/sec execute 425 sec latency 35.059 ms 2024-03-18T16:10:21.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1048190 10.32 MB/sec execute 426 sec latency 19.307 ms 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: pgmap v321: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 287 op/s 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:21.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:21 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: pgmap v321: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 287 op/s 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:21 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: pgmap v321: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 287 op/s 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:21 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1049457 10.31 MB/sec execute 427 sec latency 31.804 ms 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:22.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:22.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:22.964 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:23.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi150 to 176.2M 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi150 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi156 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:23.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:10:23.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1051618 10.31 MB/sec execute 428 sec latency 68.547 ms 2024-03-18T16:10:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:23 smithi191 ceph-mon[36784]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:10:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:23 smithi191 ceph-mon[36784]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:10:23.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:23 smithi191 ceph-mon[36784]: pgmap v322: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:23 smithi156 ceph-mon[35919]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:10:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:23 smithi156 ceph-mon[35919]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:10:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:23 smithi156 ceph-mon[35919]: pgmap v322: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:23 smithi150 ceph-mon[31946]: Adjusting osd_memory_target on smithi191 to 174.1M 2024-03-18T16:10:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:23 smithi150 ceph-mon[31946]: Unable to set osd_memory_target on smithi191 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:10:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:23 smithi150 ceph-mon[31946]: pgmap v322: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:24.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1052933 10.31 MB/sec execute 429 sec latency 82.657 ms 2024-03-18T16:10:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 -- 172.21.15.150:0/2326990299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af01012f0 msgr2=0x7f0af010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 --2- 172.21.15.150:0/2326990299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af01012f0 0x7f0af010d510 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009ca0 tx=0x7f0ae802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 -- 172.21.15.150:0/2326990299 shutdown_connections 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 --2- 172.21.15.150:0/2326990299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af01012f0 0x7f0af010d510 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 --2- 172.21.15.150:0/2326990299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af0100930 0x7f0af0100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 --2- 172.21.15.150:0/2326990299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0105680 0x7f0af0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 -- 172.21.15.150:0/2326990299 >> 172.21.15.150:0/2326990299 conn(0x7f0af0077530 msgr2=0x7f0af0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 -- 172.21.15.150:0/2326990299 shutdown_connections 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.829+0000 7f0af6537640 1 -- 172.21.15.150:0/2326990299 wait complete. 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 Processor -- start 2024-03-18T16:10:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 -- start start 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0100930 0x7f0af0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 0x7f0af0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0110ba0 con 0x7f0af0105680 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0110a20 con 0x7f0af01012f0 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af6537640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0af0110d20 con 0x7f0af0100930 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af4ad4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af4ad4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40020/0 (socket says 172.21.15.150:40020) 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0af4ad4640 1 -- 172.21.15.150:0/2907254222 learned_addr learned my addr 172.21.15.150:0/2907254222 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 0x7f0af0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0aeffff640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0100930 0x7f0af0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0aef7fe640 1 -- 172.21.15.150:0/2907254222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0100930 msgr2=0x7f0af0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0100930 0x7f0af0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.830+0000 7f0aef7fe640 1 -- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 msgr2=0x7f0af019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aef7fe640 1 -- 172.21.15.150:0/2907254222 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af019cfe0 con 0x7f0af01012f0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 0x7f0af0198530 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400e9f0 tx=0x7f0ae400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aed7fa640 1 -- 172.21.15.150:0/2907254222 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400ccc0 con 0x7f0af01012f0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af019d2d0 con 0x7f0af01012f0 2024-03-18T16:10:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aed7fa640 1 -- 172.21.15.150:0/2907254222 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae4010910 con 0x7f0af01012f0 2024-03-18T16:10:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0aed7fa640 1 -- 172.21.15.150:0/2907254222 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4018960 con 0x7f0af01012f0 2024-03-18T16:10:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.831+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af01a4b60 con 0x7f0af01012f0 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.833+0000 7f0aed7fa640 1 -- 172.21.15.150:0/2907254222 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f0ae4010ac0 con 0x7f0af01012f0 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 Processor -- start 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 -- start start 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aed7fa640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 0x7f0ad80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 0x5635e9fe8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9ff12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aed7fa640 1 -- 172.21.15.150:0/2907254222 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f0ae40cf220 con 0x7f0af01012f0 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635e9f4ca40 con 0x5635e9fe8a50 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5635e9ef5920 con 0x5635e9ff1820 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af91f6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5635e9f48ae0 con 0x5635e9fe8130 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 0x7f0ad80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 0x5635e9fe8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9ff12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 0x5635e9fe8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:52762/0 (socket says 172.21.15.150:52762) 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aeffff640 1 -- 172.21.15.150:0/1475386588 learned_addr learned my addr 172.21.15.150:0/1475386588 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1475386588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 msgr2=0x5635e9fe8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 0x5635e9fe8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1475386588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 msgr2=0x5635e9ff3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1475386588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e9f0d460 con 0x5635e9fe8a50 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 0x7f0ad80a4ed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009a60 tx=0x7f0ae8032380 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.834+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9ff12e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40cc970 tx=0x7f0ae409b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263f1c9759e0332d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0ad67fc640 1 -- 172.21.15.150:0/1475386588 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae40ccbf0 con 0x5635e9fe8a50 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0ad67fc640 1 -- 172.21.15.150:0/1475386588 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae4097fa0 con 0x5635e9fe8a50 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0ad67fc640 1 -- 172.21.15.150:0/1475386588 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4086270 con 0x5635e9fe8a50 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 -- 172.21.15.150:0/1475386588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 msgr2=0x5635e9ff12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9ff12e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40cc970 tx=0x7f0ae409b9f0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 -- 172.21.15.150:0/1475386588 shutdown_connections 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9ff12e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1475386588 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8130 0x5635e9fe8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 -- 172.21.15.150:0/1475386588 >> 172.21.15.150:0/1475386588 conn(0x5635e9fe2a50 msgr2=0x5635e9fe2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 -- 172.21.15.150:0/1475386588 shutdown_connections 2024-03-18T16:10:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.835+0000 7f0af91f6740 1 -- 172.21.15.150:0/1475386588 wait complete. 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 -- start start 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 Processor -- start 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9fef360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635e9ff43f0 con 0x5635e9fe8a50 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5635e9f08d70 con 0x5635e9fe8130 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5635e9ff4570 con 0x5635e9ff1820 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af91f6740 10 client.0 _ll_register_callbacks cb 0x5635e9c60a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:37784/0 (socket says 172.21.15.150:37784) 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aeffff640 1 -- 172.21.15.150:0/1736930224 learned_addr learned my addr 172.21.15.150:0/1736930224 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9fef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40042/0 (socket says 172.21.15.150:40042) 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 msgr2=0x5635e9fef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9fef360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1736930224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 msgr2=0x5635e9fee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aef7fe640 1 -- 172.21.15.150:0/1736930224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e9c61210 con 0x5635e9fe8a50 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.836+0000 7f0aeffff640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c6a0 tx=0x7f0ae4009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b25c8602e75aae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae407e020 con 0x5635e9fe8a50 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae4097fa0 con 0x5635e9fe8a50 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f0ae40862d0 con 0x5635e9fe8a50 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0ae40ba050 con 0x5635e9fe8a50 2024-03-18T16:10:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 10 client.16266 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635e9d2d620 con 0x5635e9fe8a50 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae4097fa0 con 0x5635e9fe8a50 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0ae4018c20 con 0x5635e9fe8a50 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 10 client.16266 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 20 client.16266 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea001860 0x5635ea003c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 4 client.16266 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.837+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635e9f59f80 con 0x5635ea001860 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.838+0000 7f0aeffff640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea001860 0x5635ea003c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.838+0000 7f0acf7fe640 10 client.16266 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.838+0000 7f0aeffff640 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea001860 0x5635ea003c50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5bc8aa7037650e1 server_cookie=3a7cb1a336f2c30e in_seq=0 out_seq=0 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0acf7fe640 1 -- 172.21.15.150:0/1736930224 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635e9f59f80 con 0x5635ea001860 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0acf7fe640 10 client.16266 handle_command_reply: tid=0 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 client.16266 shutdown 2024-03-18T16:10:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 msgr2=0x5635e9feee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c6a0 tx=0x7f0ae4009700 comp rx=0 tx=0).stop 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 shutdown_connections 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea001860 0x5635ea003c50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9fef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8a50 0x5635e9feee20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 --2- 172.21.15.150:0/1736930224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8130 0x5635e9fee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 >> 172.21.15.150:0/1736930224 conn(0x5635e9fe2a50 msgr2=0x5635e9fe47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0acf7fe640 0 client.16266 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 shutdown_connections 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 1 -- 172.21.15.150:0/1736930224 wait complete. 2024-03-18T16:10:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.839+0000 7f0af91f6740 20 client.16266 trim_cache size 0 max 16384 2024-03-18T16:10:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 Processor -- start 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 -- start start 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8a50 0x5635ea0091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01ad990 con 0x5635e9fe8130 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01ab260 con 0x5635e9ff1820 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af91f6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01a90d0 con 0x5635e9fe8a50 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40056/0 (socket says 172.21.15.150:40056) 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 -- 172.21.15.150:0/2628847892 learned_addr learned my addr 172.21.15.150:0/2628847892 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8a50 0x5635ea0091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 -- 172.21.15.150:0/2628847892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8a50 msgr2=0x5635ea0091f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8a50 0x5635ea0091f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 -- 172.21.15.150:0/2628847892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 msgr2=0x5635e9ff0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.857+0000 7f0aeffff640 1 -- 172.21.15.150:0/2628847892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e9fe76c0 con 0x5635e9fe8130 2024-03-18T16:10:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.858+0000 7f0aeffff640 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0004700 tx=0x7f0ae0012c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52eed5e430b7464 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.858+0000 7f0ad5ffb640 1 -- 172.21.15.150:0/2628847892 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0012310 con 0x5635e9fe8130 2024-03-18T16:10:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.858+0000 7f0ad5ffb640 1 -- 172.21.15.150:0/2628847892 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae000fda0 con 0x5635e9fe8130 2024-03-18T16:10:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.858+0000 7f0ad5ffb640 1 -- 172.21.15.150:0/2628847892 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00200d0 con 0x5635e9fe8130 2024-03-18T16:10:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- 172.21.15.150:0/2628847892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 msgr2=0x5635e9ffd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0004700 tx=0x7f0ae0012c30 comp rx=0 tx=0).stop 2024-03-18T16:10:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- 172.21.15.150:0/2628847892 shutdown_connections 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9ff1820 0x5635e9ff0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9fe8a50 0x5635ea0091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- 172.21.15.150:0/2628847892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ffd190 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- 172.21.15.150:0/2628847892 >> 172.21.15.150:0/2628847892 conn(0x5635e9fe2a50 msgr2=0x5635e9fe9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- 172.21.15.150:0/2628847892 shutdown_connections 2024-03-18T16:10:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- 172.21.15.150:0/2628847892 wait complete. 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- start start 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 Processor -- start 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 0x5635e9ff5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9ff5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01a9720 con 0x5635e9fe8130 2024-03-18T16:10:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01aab90 con 0x5635e9fe8a50 2024-03-18T16:10:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.859+0000 7f0af91f6740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f0af01a9df0 con 0x5635e9ff1820 2024-03-18T16:10:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0af91f6740 10 client.0 _ll_register_callbacks cb 0x7f0ae0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40062/0 (socket says 172.21.15.150:40062) 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 -- 172.21.15.150:0/316016515 learned_addr learned my addr 172.21.15.150:0/316016515 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 0x5635e9ff5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0af4ad4640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9ff5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 -- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 msgr2=0x5635e9ff5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9ff5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 -- 172.21.15.150:0/316016515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 msgr2=0x5635e9ff5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 0x5635e9ff5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 -- 172.21.15.150:0/316016515 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635ea066820 con 0x5635e9fe8130 2024-03-18T16:10:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 0x5635e9ff5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0aeffff640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0010b80 tx=0x7f0ae00109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2753ba9636af5cff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0acf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0004050 con 0x5635e9fe8130 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f0ae00041f0 con 0x5635e9fe8130 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae000fda0 con 0x5635e9fe8130 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0ae00043f0 con 0x5635e9fe8130 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0af91f6740 10 client.16272 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.860+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635ea067a40 con 0x5635e9fe8130 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae000fda0 con 0x5635e9fe8130 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0ae0020660 con 0x5635e9fe8130 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 10 client.16272 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 20 client.16272 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea068080 0x5635ea06a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 4 client.16272 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:10:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635e9f542d0 con 0x5635ea068080 2024-03-18T16:10:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea068080 0x5635ea06a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0acf7fe640 10 client.16272 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:24.877 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.861+0000 7f0aef7fe640 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea068080 0x5635ea06a4a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6127d325f690e53d server_cookie=b3d60ce7847cf713 in_seq=0 out_seq=0 2024-03-18T16:10:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.872+0000 7f0acf7fe640 1 -- 172.21.15.150:0/316016515 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635e9f542d0 con 0x5635ea068080 2024-03-18T16:10:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.872+0000 7f0acf7fe640 10 client.16272 handle_command_reply: tid=0 2024-03-18T16:10:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.872+0000 7f0af91f6740 1 client.16272 shutdown 2024-03-18T16:10:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.872+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 msgr2=0x5635e9ff4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.872+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0010b80 tx=0x7f0ae00109e0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 shutdown_connections 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635ea068080 0x5635ea06a4a0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635e9ff1820 0x5635e9ff5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635e9fe8a50 0x5635e9ff5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0acf7fe640 0 client.16272 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 --2- 172.21.15.150:0/316016515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635e9fe8130 0x5635e9ff4fe0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 >> 172.21.15.150:0/316016515 conn(0x5635e9fe2a50 msgr2=0x5635e9fe3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 shutdown_connections 2024-03-18T16:10:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 1 -- 172.21.15.150:0/316016515 wait complete. 2024-03-18T16:10:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af91f6740 20 client.16272 trim_cache size 0 max 16384 2024-03-18T16:10:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:10:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:10:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:10:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:10:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 msgr2=0x7f0ad80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 0x7f0ad80a4ed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009a60 tx=0x7f0ae8032380 comp rx=0 tx=0).stop 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 msgr2=0x7f0af0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 0x7f0af0198530 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400e9f0 tx=0x7f0ae400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 shutdown_connections 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f0ad80a2a10 0x7f0ad80a4ed0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.885 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0af0105680 0x7f0af019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0af01012f0 0x7f0af0198530 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 --2- 172.21.15.150:0/2907254222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0af0100930 0x7f0af0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 >> 172.21.15.150:0/2907254222 conn(0x7f0af0077530 msgr2=0x7f0af00690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 shutdown_connections 2024-03-18T16:10:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:24.873+0000 7f0af6537640 1 -- 172.21.15.150:0/2907254222 wait complete. 2024-03-18T16:10:25.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1054878 10.30 MB/sec execute 430 sec latency 33.322 ms 2024-03-18T16:10:25.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:25 smithi191 ceph-mon[36784]: pgmap v323: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:25.768 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:25 smithi156 ceph-mon[35919]: pgmap v323: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:25.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:25 smithi150 ceph-mon[31946]: pgmap v323: 97 pgs: 97 active+clean; 608 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:10:26.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1056481 10.30 MB/sec execute 431 sec latency 17.540 ms 2024-03-18T16:10:27.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1058583 10.31 MB/sec execute 432 sec latency 21.729 ms 2024-03-18T16:10:27.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:27 smithi191 ceph-mon[36784]: pgmap v324: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 274 op/s 2024-03-18T16:10:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:27 smithi156 ceph-mon[35919]: pgmap v324: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 274 op/s 2024-03-18T16:10:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:27 smithi150 ceph-mon[31946]: pgmap v324: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 274 op/s 2024-03-18T16:10:28.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1059821 10.29 MB/sec execute 433 sec latency 38.222 ms 2024-03-18T16:10:29.340 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1061804 10.29 MB/sec execute 434 sec latency 57.233 ms 2024-03-18T16:10:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:29 smithi191 ceph-mon[36784]: pgmap v325: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-03-18T16:10:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:29 smithi156 ceph-mon[35919]: pgmap v325: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-03-18T16:10:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:29 smithi150 ceph-mon[31946]: pgmap v325: 97 pgs: 97 active+clean; 622 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-03-18T16:10:30.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1063365 10.29 MB/sec execute 435 sec latency 104.301 ms 2024-03-18T16:10:31.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1065160 10.29 MB/sec execute 436 sec latency 36.720 ms 2024-03-18T16:10:31.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:31 smithi191 ceph-mon[36784]: pgmap v326: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:31 smithi156 ceph-mon[35919]: pgmap v326: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:31 smithi150 ceph-mon[31946]: pgmap v326: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:32.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1066625 10.28 MB/sec execute 437 sec latency 54.215 ms 2024-03-18T16:10:33.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1068662 10.29 MB/sec execute 438 sec latency 21.709 ms 2024-03-18T16:10:33.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:33 smithi191 ceph-mon[36784]: pgmap v327: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:33 smithi156 ceph-mon[35919]: pgmap v327: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:33 smithi150 ceph-mon[31946]: pgmap v327: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:34.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1070364 10.28 MB/sec execute 439 sec latency 26.728 ms 2024-03-18T16:10:35.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1072130 10.28 MB/sec execute 440 sec latency 84.291 ms 2024-03-18T16:10:35.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:35 smithi191 ceph-mon[36784]: pgmap v328: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:35 smithi156 ceph-mon[35919]: pgmap v328: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:35 smithi150 ceph-mon[31946]: pgmap v328: 97 pgs: 97 active+clean; 629 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-03-18T16:10:36.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1073760 10.28 MB/sec execute 441 sec latency 22.564 ms 2024-03-18T16:10:37.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1075680 10.28 MB/sec execute 442 sec latency 32.005 ms 2024-03-18T16:10:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:37 smithi191 ceph-mon[36784]: pgmap v329: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 -- 172.21.15.150:0/2813758393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 msgr2=0x7f7f8010d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/2813758393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f8010d580 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009930 tx=0x7f7f7802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 -- 172.21.15.150:0/2813758393 shutdown_connections 2024-03-18T16:10:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/2813758393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f8010d580 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/2813758393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801006b0 0x7f7f80100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/2813758393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801066c0 0x7f7f80106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 -- 172.21.15.150:0/2813758393 >> 172.21.15.150:0/2813758393 conn(0x7f7f80077810 msgr2=0x7f7f80077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 -- 172.21.15.150:0/2813758393 shutdown_connections 2024-03-18T16:10:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.964+0000 7f7f84d0a640 1 -- 172.21.15.150:0/2813758393 wait complete. 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 Processor -- start 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 -- start start 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f80198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 0x7f7f8019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7f80110ad0 con 0x7f7f80101070 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f80110950 con 0x7f7f801066c0 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f84d0a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7f80110c50 con 0x7f7f801006b0 2024-03-18T16:10:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.965+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f80198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:55678/0 (socket says 172.21.15.150:55678) 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7e59c640 1 -- 172.21.15.150:0/3838979422 learned_addr learned my addr 172.21.15.150:0/3838979422 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 0x7f7f8019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3838979422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 msgr2=0x7f7f80198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3838979422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 msgr2=0x7f7f8019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 0x7f7f8019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3838979422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8019cff0 con 0x7f7f80101070 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 0x7f7f8019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f80198590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400e9f0 tx=0x7f7f7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f6f7fe640 1 -- 172.21.15.150:0/3838979422 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7400ccc0 con 0x7f7f80101070 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f6f7fe640 1 -- 172.21.15.150:0/3838979422 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74010910 con 0x7f7f80101070 2024-03-18T16:10:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8019d2e0 con 0x7f7f80101070 2024-03-18T16:10:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f6f7fe640 1 -- 172.21.15.150:0/3838979422 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74018960 con 0x7f7f80101070 2024-03-18T16:10:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.966+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f801a4b70 con 0x7f7f80101070 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.968+0000 7f7f6f7fe640 1 -- 172.21.15.150:0/3838979422 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f7f74010ac0 con 0x7f7f80101070 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 Processor -- start 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f6f7fe640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 0x7f7f680a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 0x7f7f680a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f6f7fe640 1 -- 172.21.15.150:0/3838979422 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f7f740cf220 con 0x7f7f80101070 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 -- start start 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 0x55dd3e7712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.977 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e773c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e6cca40 con 0x55dd3e768130 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e675920 con 0x55dd3e768a50 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.969+0000 7f7f879cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e6c8ae0 con 0x55dd3e771820 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49438/0 (socket says 172.21.15.150:49438) 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 -- 172.21.15.150:0/3216302800 learned_addr learned my addr 172.21.15.150:0/3216302800 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 0x55dd3e7712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 0x7f7f680a4ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f78005940 tx=0x7f7f780058d0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e773c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 -- 172.21.15.150:0/3216302800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 msgr2=0x55dd3e773c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e773c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 -- 172.21.15.150:0/3216302800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 msgr2=0x55dd3e7712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 0x55dd3e7712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 -- 172.21.15.150:0/3216302800 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3e68d460 con 0x55dd3e768130 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 0x55dd3e7712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e773c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f700118d0 tx=0x7f7f70011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5790f2fc7c76fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f6cff9640 1 -- 172.21.15.150:0/3216302800 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7001e070 con 0x55dd3e768130 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f6cff9640 1 -- 172.21.15.150:0/3216302800 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7000fe80 con 0x55dd3e768130 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.970+0000 7f7f6cff9640 1 -- 172.21.15.150:0/3216302800 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70019e20 con 0x55dd3e768130 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- 172.21.15.150:0/3216302800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 msgr2=0x55dd3e768510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f700118d0 tx=0x7f7f70011d90 comp rx=0 tx=0).stop 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- 172.21.15.150:0/3216302800 shutdown_connections 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e773c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768a50 0x55dd3e7712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3216302800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768130 0x55dd3e768510 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- 172.21.15.150:0/3216302800 >> 172.21.15.150:0/3216302800 conn(0x55dd3e762a50 msgr2=0x55dd3e762e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- 172.21.15.150:0/3216302800 shutdown_connections 2024-03-18T16:10:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- 172.21.15.150:0/3216302800 wait complete. 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 -- start start 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.971+0000 7f7f879cd740 1 Processor -- start 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e76e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e76f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e774450 con 0x55dd3e768a50 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e688d70 con 0x55dd3e768130 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55dd3e7745d0 con 0x55dd3e771820 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 10 client.0 _ll_register_callbacks cb 0x55dd3e3e0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e76e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7ed9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e76f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49442/0 (socket says 172.21.15.150:49442) 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e76e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:34106/0 (socket says 172.21.15.150:34106) 2024-03-18T16:10:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/293577521 learned_addr learned my addr 172.21.15.150:0/293577521 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 msgr2=0x55dd3e76f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e76f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/293577521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 msgr2=0x55dd3e76e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e76e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/293577521 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd3e3e1210 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f740cca20 tx=0x7f7f7409f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bce3ab0b1a5f8c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f740b4060 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f7f740a3d90 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.972+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dd3e6d9f80 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f7409cbb0 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 10 client.16284 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd3e4ad620 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dd3e6d9f80 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f7f74018b00 con 0x55dd3e768a50 2024-03-18T16:10:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 10 client.16284 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 20 client.16284 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e781890 0x55dd3e783c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 4 client.16284 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd3e6d42d0 con 0x55dd3e781890 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.973+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e781890 0x55dd3e783c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e781890 0x55dd3e783c80 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3703c89fd6c299fb server_cookie=d55a46295e02efd5 in_seq=0 out_seq=0 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f65ffb640 10 client.16284 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f65ffb640 1 -- 172.21.15.150:0/293577521 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd3e6d42d0 con 0x55dd3e781890 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f65ffb640 10 client.16284 handle_command_reply: tid=0 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 client.16284 shutdown 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 msgr2=0x55dd3e76ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f740cca20 tx=0x7f7f7409f9d0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 shutdown_connections 2024-03-18T16:10:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e781890 0x55dd3e783c80 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e76f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f65ffb640 0 client.16284 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e76ece0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.974+0000 7f7f879cd740 1 --2- 172.21.15.150:0/293577521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e76e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.975+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 >> 172.21.15.150:0/293577521 conn(0x55dd3e762a50 msgr2=0x55dd3e7647a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.975+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 shutdown_connections 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.975+0000 7f7f879cd740 1 -- 172.21.15.150:0/293577521 wait complete. 2024-03-18T16:10:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.975+0000 7f7f879cd740 20 client.16284 trim_cache size 0 max 16384 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 Processor -- start 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 -- start start 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e770ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e75da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801afd00 con 0x55dd3e768a50 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801ad5d0 con 0x55dd3e768130 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.989+0000 7f7f879cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801ab440 con 0x55dd3e771820 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.992 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49452/0 (socket says 172.21.15.150:49452) 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3840497290 learned_addr learned my addr 172.21.15.150:0/3840497290 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e75da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e770ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3840497290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 msgr2=0x55dd3e75da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e75da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3840497290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 msgr2=0x55dd3e770ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e770ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 -- 172.21.15.150:0/3840497290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd3e76d750 con 0x55dd3e768a50 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f7dd9b640 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7f74099be0 tx=0x7f7f7400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad977082ac83cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f67fff640 1 -- 172.21.15.150:0/3840497290 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f740b4040 con 0x55dd3e768a50 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f67fff640 1 -- 172.21.15.150:0/3840497290 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74018b00 con 0x55dd3e768a50 2024-03-18T16:10:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.990+0000 7f7f67fff640 1 -- 172.21.15.150:0/3840497290 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7409ad80 con 0x55dd3e768a50 2024-03-18T16:10:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- 172.21.15.150:0/3840497290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 msgr2=0x55dd3e77d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7f74099be0 tx=0x7f7f7400cf60 comp rx=0 tx=0).stop 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- 172.21.15.150:0/3840497290 shutdown_connections 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e771820 0x55dd3e75da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e768a50 0x55dd3e77d1f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- 172.21.15.150:0/3840497290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e770ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- 172.21.15.150:0/3840497290 >> 172.21.15.150:0/3840497290 conn(0x55dd3e762a50 msgr2=0x55dd3e76eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- 172.21.15.150:0/3840497290 shutdown_connections 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- 172.21.15.150:0/3840497290 wait complete. 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 -- start start 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 Processor -- start 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e7e0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e768a50 0x55dd3e7e13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.991+0000 7f7f879cd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801aba90 con 0x55dd3e771820 2024-03-18T16:10:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801acf00 con 0x55dd3e768130 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801ac160 con 0x55dd3e768a50 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 10 client.0 _ll_register_callbacks cb 0x7f7f7409b360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7dd9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e768a50 0x55dd3e7e13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49466/0 (socket says 172.21.15.150:49466) 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e7e0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 -- 172.21.15.150:0/1898205453 learned_addr learned my addr 172.21.15.150:0/1898205453 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 -- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e768a50 msgr2=0x55dd3e7e13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e768a50 0x55dd3e7e13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 -- 172.21.15.150:0/1898205453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 msgr2=0x55dd3e7e0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e7e0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 -- 172.21.15.150:0/1898205453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd3e775f40 con 0x55dd3e771820 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e7e0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f7ed9d640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009930 tx=0x7f7f7802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d16182376bde6b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f65ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7805a060 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7f7f78005480 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f74018b00 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f780035f0 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 10 client.16290 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.992+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd3e7e85c0 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74018b00 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f7f7805e560 con 0x55dd3e771820 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 10 client.16290 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 10 client.16290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 20 client.16290 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e7e8ce0 0x55dd3e7eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 4 client.16290 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:10:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd3e652770 con 0x55dd3e7e8ce0 2024-03-18T16:10:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e7e8ce0 0x55dd3e7eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f65ffb640 10 client.16290 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:38.000 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:37.993+0000 7f7f7e59c640 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e7e8ce0 0x55dd3e7eb0d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=961cb402146b7419 server_cookie=1a9434d485e05fd7 in_seq=0 out_seq=0 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.004+0000 7f7f65ffb640 1 -- 172.21.15.150:0/1898205453 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd3e652770 con 0x55dd3e7e8ce0 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.004+0000 7f7f65ffb640 10 client.16290 handle_command_reply: tid=0 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 client.16290 shutdown 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 msgr2=0x55dd3e775880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009930 tx=0x7f7f7802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 shutdown_connections 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55dd3e7e8ce0 0x55dd3e7eb0d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dd3e771820 0x55dd3e775880 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55dd3e768a50 0x55dd3e7e13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 --2- 172.21.15.150:0/1898205453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd3e768130 0x55dd3e7e0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f65ffb640 0 client.16290 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 >> 172.21.15.150:0/1898205453 conn(0x55dd3e762a50 msgr2=0x55dd3e761320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 shutdown_connections 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 1 -- 172.21.15.150:0/1898205453 wait complete. 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f879cd740 20 client.16290 trim_cache size 0 max 16384 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:10:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:10:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 msgr2=0x7f7f680a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 0x7f7f680a4ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f78005940 tx=0x7f7f780058d0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 msgr2=0x7f7f80198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f80198590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400e9f0 tx=0x7f7f7400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 shutdown_connections 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f7f680a2a10 0x7f7f680a4ed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f801066c0 0x7f7f8019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80101070 0x7f7f80198590 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 --2- 172.21.15.150:0/3838979422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f801006b0 0x7f7f80198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.005+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 >> 172.21.15.150:0/3838979422 conn(0x7f7f80077810 msgr2=0x7f7f8006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.006+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 shutdown_connections 2024-03-18T16:10:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:38.006+0000 7f7f84d0a640 1 -- 172.21.15.150:0/3838979422 wait complete. 2024-03-18T16:10:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:37 smithi156 ceph-mon[35919]: pgmap v329: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:37 smithi150 ceph-mon[31946]: pgmap v329: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:38.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1077296 10.27 MB/sec execute 443 sec latency 17.832 ms 2024-03-18T16:10:39.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1079317 10.28 MB/sec execute 444 sec latency 61.546 ms 2024-03-18T16:10:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:39 smithi191 ceph-mon[36784]: pgmap v330: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:40.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:39 smithi156 ceph-mon[35919]: pgmap v330: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:39 smithi150 ceph-mon[31946]: pgmap v330: 97 pgs: 97 active+clean; 593 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 177 op/s 2024-03-18T16:10:40.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1080738 10.27 MB/sec execute 445 sec latency 62.832 ms 2024-03-18T16:10:41.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1082793 10.27 MB/sec execute 446 sec latency 20.015 ms 2024-03-18T16:10:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:41 smithi191 ceph-mon[36784]: pgmap v331: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-03-18T16:10:42.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:41 smithi156 ceph-mon[35919]: pgmap v331: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-03-18T16:10:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:41 smithi150 ceph-mon[31946]: pgmap v331: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 265 op/s 2024-03-18T16:10:42.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1084399 10.27 MB/sec execute 447 sec latency 14.524 ms 2024-03-18T16:10:43.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1086430 10.27 MB/sec execute 448 sec latency 20.653 ms 2024-03-18T16:10:43.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:43 smithi191 ceph-mon[36784]: pgmap v332: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:43 smithi156 ceph-mon[35919]: pgmap v332: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:43 smithi150 ceph-mon[31946]: pgmap v332: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:44.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1088020 10.27 MB/sec execute 449 sec latency 38.321 ms 2024-03-18T16:10:45.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1089563 10.26 MB/sec execute 450 sec latency 161.354 ms 2024-03-18T16:10:45.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:45 smithi191 ceph-mon[36784]: pgmap v333: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:45 smithi156 ceph-mon[35919]: pgmap v333: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:46.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:45 smithi150 ceph-mon[31946]: pgmap v333: 97 pgs: 97 active+clean; 606 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 177 op/s 2024-03-18T16:10:46.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1091295 10.26 MB/sec execute 451 sec latency 12.847 ms 2024-03-18T16:10:47.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1093272 10.26 MB/sec execute 452 sec latency 58.231 ms 2024-03-18T16:10:47.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:47 smithi191 ceph-mon[36784]: pgmap v334: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-18T16:10:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:47 smithi156 ceph-mon[35919]: pgmap v334: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-18T16:10:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:47 smithi150 ceph-mon[31946]: pgmap v334: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 267 op/s 2024-03-18T16:10:48.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1094928 10.26 MB/sec execute 453 sec latency 33.710 ms 2024-03-18T16:10:49.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1096967 10.26 MB/sec execute 454 sec latency 21.346 ms 2024-03-18T16:10:49.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:49 smithi191 ceph-mon[36784]: pgmap v335: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-03-18T16:10:50.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:49 smithi156 ceph-mon[35919]: pgmap v335: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-03-18T16:10:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:49 smithi150 ceph-mon[31946]: pgmap v335: 97 pgs: 97 active+clean; 614 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-03-18T16:10:50.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1098220 10.25 MB/sec execute 455 sec latency 45.243 ms 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 -- 172.21.15.150:0/2221997982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff714102170 msgr2=0x7ff71410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 --2- 172.21.15.150:0/2221997982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff714102170 0x7ff71410e680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009cd0 tx=0x7ff70c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 -- 172.21.15.150:0/2221997982 shutdown_connections 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 --2- 172.21.15.150:0/2221997982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff714102170 0x7ff71410e680 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 --2- 172.21.15.150:0/2221997982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 --2- 172.21.15.150:0/2221997982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7141077c0 0x7ff714107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.096 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 -- 172.21.15.150:0/2221997982 >> 172.21.15.150:0/2221997982 conn(0x7ff7140fd2c0 msgr2=0x7ff7140ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 -- 172.21.15.150:0/2221997982 shutdown_connections 2024-03-18T16:10:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.093+0000 7ff7190dc640 1 -- 172.21.15.150:0/2221997982 wait complete. 2024-03-18T16:10:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 Processor -- start 2024-03-18T16:10:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 -- start start 2024-03-18T16:10:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 0x7ff714198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 0x7ff71419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff714111bd0 con 0x7ff7141077c0 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff714111a50 con 0x7ff714102170 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff7190dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff714111d50 con 0x7ff7141017b0 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 0x7ff714198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:41750/0 (socket says 172.21.15.150:41750) 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 0x7ff71419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff712d9d640 1 -- 172.21.15.150:0/1733563044 learned_addr learned my addr 172.21.15.150:0/1733563044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 -- 172.21.15.150:0/1733563044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 msgr2=0x7ff714197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 -- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 msgr2=0x7ff71419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 0x7ff71419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.094+0000 7ff71259c640 1 -- 172.21.15.150:0/1733563044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71419cf70 con 0x7ff714102170 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff712d9d640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff71359e640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 0x7ff71419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff71259c640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 0x7ff714198510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e9f0 tx=0x7ff70800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff6fbfff640 1 -- 172.21.15.150:0/1733563044 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70800ccc0 con 0x7ff714102170 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71419d260 con 0x7ff714102170 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff6fbfff640 1 -- 172.21.15.150:0/1733563044 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff708010910 con 0x7ff714102170 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7141a4af0 con 0x7ff714102170 2024-03-18T16:10:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.095+0000 7ff6fbfff640 1 -- 172.21.15.150:0/1733563044 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708018960 con 0x7ff714102170 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.097+0000 7ff6fbfff640 1 -- 172.21.15.150:0/1733563044 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7ff708010ac0 con 0x7ff714102170 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.097+0000 7ff71bda2740 1 Processor -- start 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.097+0000 7ff71bda2740 1 -- start start 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503a130 0x5635f503a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f5045c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635f4f9ea40 con 0x5635f503aa50 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5635f4f47920 con 0x5635f503a130 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71bda2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5635f4f9aae0 con 0x5635f5043820 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48898/0 (socket says 172.21.15.150:48898) 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 -- 172.21.15.150:0/2113998382 learned_addr learned my addr 172.21.15.150:0/2113998382 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71359e640 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503a130 0x5635f503a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff6fbfff640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 0x7ff6fc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71259c640 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f5045c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71359e640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 0x7ff6fc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 -- 172.21.15.150:0/2113998382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 msgr2=0x5635f5045c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f5045c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 -- 172.21.15.150:0/2113998382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503a130 msgr2=0x5635f503a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503a130 0x5635f503a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 -- 172.21.15.150:0/2113998382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635f4f5f460 con 0x5635f503aa50 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff6fbfff640 1 -- 172.21.15.150:0/1733563044 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7ff7080cf280 con 0x7ff714102170 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff712d9d640 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff7040115c0 tx=0x7ff704011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e98cc22c8a2b538 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff6f97fa640 1 -- 172.21.15.150:0/2113998382 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70400f660 con 0x5635f503aa50 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.098+0000 7ff71359e640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 0x7ff6fc0a4ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff70c005cc0 tx=0x7ff70c02c0f0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff6f97fa640 1 -- 172.21.15.150:0/2113998382 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70400fce0 con 0x5635f503aa50 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff6f97fa640 1 -- 172.21.15.150:0/2113998382 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704019d80 con 0x5635f503aa50 2024-03-18T16:10:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- 172.21.15.150:0/2113998382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 msgr2=0x5635f50432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff7040115c0 tx=0x7ff704011a80 comp rx=0 tx=0).stop 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- 172.21.15.150:0/2113998382 shutdown_connections 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f5045c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f50432e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 --2- 172.21.15.150:0/2113998382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503a130 0x5635f503a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- 172.21.15.150:0/2113998382 >> 172.21.15.150:0/2113998382 conn(0x5635f5034a50 msgr2=0x5635f5034e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- 172.21.15.150:0/2113998382 shutdown_connections 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- 172.21.15.150:0/2113998382 wait complete. 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.099+0000 7ff71bda2740 1 -- start start 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 Processor -- start 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f503a130 0x5635f50408e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f5043820 0x5635f5041360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635f5046450 con 0x5635f503aa50 2024-03-18T16:10:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5635f4f5ad70 con 0x5635f5043820 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5635f50465d0 con 0x5635f503a130 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 10 client.0 _ll_register_callbacks cb 0x5635f4cb2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48908/0 (socket says 172.21.15.150:48908) 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 -- 172.21.15.150:0/3597748457 learned_addr learned my addr 172.21.15.150:0/3597748457 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff712d9d640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f503a130 0x5635f50408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71359e640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f5043820 0x5635f5041360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 -- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f503a130 msgr2=0x5635f50408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f503a130 0x5635f50408e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 -- 172.21.15.150:0/3597748457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f5043820 msgr2=0x5635f5041360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f5043820 0x5635f5041360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 -- 172.21.15.150:0/3597748457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635f4cb3210 con 0x5635f503aa50 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71259c640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff708098300 tx=0x7ff708009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786ed63207a7443b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff6f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7080862e0 con 0x5635f503aa50 2024-03-18T16:10:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7ff70809cbb0 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.100+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff70400fce0 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff70809cdb0 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 10 client.16302 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635f4d7f620 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff70400fce0 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff708018c20 con 0x5635f503aa50 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 10 client.16302 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 10 client.16302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 20 client.16302 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50539d0 0x5635f5055dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 4 client.16302 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635f4fabf80 con 0x5635f50539d0 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.101+0000 7ff712d9d640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50539d0 0x5635f5055dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.102+0000 7ff712d9d640 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50539d0 0x5635f5055dc0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d0a07ddc6fe8ef server_cookie=53782ff221480e2b in_seq=0 out_seq=0 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.102+0000 7ff6f6ff5640 10 client.16302 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:51.110 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.102+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/3597748457 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635f4fabf80 con 0x5635f50539d0 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.102+0000 7ff6f6ff5640 10 client.16302 handle_command_reply: tid=0 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.102+0000 7ff71bda2740 1 client.16302 shutdown 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 msgr2=0x5635f5040e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff708098300 tx=0x7ff708009700 comp rx=0 tx=0).stop 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 shutdown_connections 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50539d0 0x5635f5055dc0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f5043820 0x5635f5041360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503aa50 0x5635f5040e20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff6f6ff5640 0 client.16302 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 --2- 172.21.15.150:0/3597748457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f503a130 0x5635f50408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 >> 172.21.15.150:0/3597748457 conn(0x5635f5034a50 msgr2=0x5635f50367a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 shutdown_connections 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 1 -- 172.21.15.150:0/3597748457 wait complete. 2024-03-18T16:10:51.111 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.103+0000 7ff71bda2740 20 client.16302 trim_cache size 0 max 16384 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.117+0000 7ff71bda2740 1 Processor -- start 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.117+0000 7ff71bda2740 1 -- start start 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.117+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.117+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f5042710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f505b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71bda2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141afe10 con 0x5635f503a130 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71bda2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141ad6e0 con 0x5635f503aa50 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71bda2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141ab550 con 0x5635f5043820 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.121 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48918/0 (socket says 172.21.15.150:48918) 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f5042710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff712d9d640 1 -- 172.21.15.150:0/4259160307 learned_addr learned my addr 172.21.15.150:0/4259160307 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71359e640 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f505b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 -- 172.21.15.150:0/4259160307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 msgr2=0x5635f505b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f505b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 -- 172.21.15.150:0/4259160307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 msgr2=0x5635f504f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 -- 172.21.15.150:0/4259160307 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635f5035620 con 0x5635f503aa50 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff712d9d640 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff71259c640 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f5042710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff70809b630 tx=0x7ff708098220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5699bfe7b5a6907e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff6f8ff9640 1 -- 172.21.15.150:0/4259160307 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70809f9d0 con 0x5635f503aa50 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff6f8ff9640 1 -- 172.21.15.150:0/4259160307 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff708086110 con 0x5635f503aa50 2024-03-18T16:10:51.122 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.118+0000 7ff6f8ff9640 1 -- 172.21.15.150:0/4259160307 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70809aa50 con 0x5635f503aa50 2024-03-18T16:10:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 -- 172.21.15.150:0/4259160307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 msgr2=0x5635f5042710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f5042710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff70809b630 tx=0x7ff708098220 comp rx=0 tx=0).stop 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 -- 172.21.15.150:0/4259160307 shutdown_connections 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f505b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f5042710 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 --2- 172.21.15.150:0/4259160307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f504f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 -- 172.21.15.150:0/4259160307 >> 172.21.15.150:0/4259160307 conn(0x5635f5034a50 msgr2=0x5635f5042000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 -- 172.21.15.150:0/4259160307 shutdown_connections 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.119+0000 7ff71bda2740 1 -- 172.21.15.150:0/4259160307 wait complete. 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 -- start start 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 Processor -- start 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f503db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f503e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141abba0 con 0x5635f503a130 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141ad010 con 0x5635f503aa50 2024-03-18T16:10:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7ff7141ac270 con 0x5635f5043820 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f503db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71bda2740 10 client.0 _ll_register_callbacks cb 0x7ff70800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48934/0 (socket says 172.21.15.150:48934) 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f503db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33050/0 (socket says 172.21.15.150:33050) 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 -- 172.21.15.150:0/13290903 learned_addr learned my addr 172.21.15.150:0/13290903 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff71359e640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f503e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 -- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 msgr2=0x5635f503e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f503e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 -- 172.21.15.150:0/13290903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 msgr2=0x5635f503db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f503db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.120+0000 7ff712d9d640 1 -- 172.21.15.150:0/13290903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635f50bd220 con 0x5635f503a130 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff712d9d640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff704009250 tx=0x7ff704009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30c51054f08a510 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704019740 con 0x5635f503a130 2024-03-18T16:10:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8527+0+0 (secure 0 0 0) 0x7ff704019a30 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff708086110 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff704019c30 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 10 client.16308 fetch_fsmap learned FSMap version 31 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635f50be540 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff708086110 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff70402c070 con 0x5635f503a130 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 10 client.16308 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 20 client.16308 populate_metadata read hostname 'smithi150' 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50beb00 0x5635f50c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 4 client.16308 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.121+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635f4fa62d0 con 0x5635f50beb00 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.122+0000 7ff71259c640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50beb00 0x5635f50c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.122+0000 7ff71259c640 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50beb00 0x5635f50c0f20 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9767e6ecef366e5b server_cookie=c7968050b52da5fc in_seq=0 out_seq=0 2024-03-18T16:10:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.122+0000 7ff6f6ff5640 10 client.16308 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff6f6ff5640 1 -- 172.21.15.150:0/13290903 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635f4fa62d0 con 0x5635f50beb00 2024-03-18T16:10:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff6f6ff5640 10 client.16308 handle_command_reply: tid=0 2024-03-18T16:10:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 client.16308 shutdown 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 msgr2=0x5635f503d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff704009250 tx=0x7ff704009530 comp rx=0 tx=0).stop 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 shutdown_connections 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5635f50beb00 0x5635f50c0f20 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5635f5043820 0x5635f503e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5635f503aa50 0x5635f503db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 --2- 172.21.15.150:0/13290903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635f503a130 0x5635f503d5e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 >> 172.21.15.150:0/13290903 conn(0x5635f5034a50 msgr2=0x5635f5033bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.138+0000 7ff6f6ff5640 0 client.16308 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.139+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 shutdown_connections 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.139+0000 7ff71bda2740 1 -- 172.21.15.150:0/13290903 wait complete. 2024-03-18T16:10:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.139+0000 7ff71bda2740 20 client.16308 trim_cache size 0 max 16384 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 msgr2=0x7ff6fc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 0x7ff6fc0a4ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff70c005cc0 tx=0x7ff70c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 msgr2=0x7ff714198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 0x7ff714198510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e9f0 tx=0x7ff70800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 shutdown_connections 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7ff6fc0a2a10 0x7ff6fc0a4ed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7141077c0 0x7ff71419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff714102170 0x7ff714198510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 --2- 172.21.15.150:0/1733563044 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff7141017b0 0x7ff714197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 >> 172.21.15.150:0/1733563044 conn(0x7ff7140fd2c0 msgr2=0x7ff7140fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 shutdown_connections 2024-03-18T16:10:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:10:51.140+0000 7ff7190dc640 1 -- 172.21.15.150:0/1733563044 wait complete. 2024-03-18T16:10:51.341 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1099990 10.24 MB/sec execute 456 sec latency 29.080 ms 2024-03-18T16:10:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:51 smithi191 ceph-mon[36784]: pgmap v336: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 267 op/s 2024-03-18T16:10:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:51 smithi156 ceph-mon[35919]: pgmap v336: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 267 op/s 2024-03-18T16:10:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:51 smithi150 ceph-mon[31946]: pgmap v336: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 267 op/s 2024-03-18T16:10:52.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1101708 10.25 MB/sec execute 457 sec latency 10.740 ms 2024-03-18T16:10:53.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1103490 10.24 MB/sec execute 458 sec latency 24.170 ms 2024-03-18T16:10:53.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:53 smithi191 ceph-mon[36784]: pgmap v337: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:54.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:53 smithi156 ceph-mon[35919]: pgmap v337: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:53 smithi150 ceph-mon[31946]: pgmap v337: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:54.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1105289 10.25 MB/sec execute 459 sec latency 17.444 ms 2024-03-18T16:10:55.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1106609 10.24 MB/sec execute 460 sec latency 114.957 ms 2024-03-18T16:10:55.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:55 smithi191 ceph-mon[36784]: pgmap v338: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:56.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:55 smithi156 ceph-mon[35919]: pgmap v338: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:55 smithi150 ceph-mon[31946]: pgmap v338: 97 pgs: 97 active+clean; 626 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 179 op/s 2024-03-18T16:10:56.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1108670 10.24 MB/sec execute 461 sec latency 17.927 ms 2024-03-18T16:10:57.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1109797 10.23 MB/sec execute 462 sec latency 33.629 ms 2024-03-18T16:10:57.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:57 smithi191 ceph-mon[36784]: pgmap v339: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:58.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:57 smithi156 ceph-mon[35919]: pgmap v339: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:57 smithi150 ceph-mon[31946]: pgmap v339: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.9 MiB/s wr, 269 op/s 2024-03-18T16:10:58.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1112005 10.24 MB/sec execute 463 sec latency 19.785 ms 2024-03-18T16:10:59.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1113346 10.23 MB/sec execute 464 sec latency 30.795 ms 2024-03-18T16:11:00.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:10:59 smithi156 ceph-mon[35919]: pgmap v340: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:11:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:10:59 smithi150 ceph-mon[31946]: pgmap v340: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:11:00.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:10:59 smithi191 ceph-mon[36784]: pgmap v340: 97 pgs: 97 active+clean; 635 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-03-18T16:11:00.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1115405 10.23 MB/sec execute 465 sec latency 55.247 ms 2024-03-18T16:11:01.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1116134 10.22 MB/sec execute 466 sec latency 383.921 ms 2024-03-18T16:11:02.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:01 smithi156 ceph-mon[35919]: pgmap v341: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 270 op/s 2024-03-18T16:11:02.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:01 smithi150 ceph-mon[31946]: pgmap v341: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 270 op/s 2024-03-18T16:11:02.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:01 smithi191 ceph-mon[36784]: pgmap v341: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 270 op/s 2024-03-18T16:11:02.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1117307 10.20 MB/sec execute 467 sec latency 24.015 ms 2024-03-18T16:11:03.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1119278 10.21 MB/sec execute 468 sec latency 26.958 ms 2024-03-18T16:11:04.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:03 smithi156 ceph-mon[35919]: pgmap v342: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:11:04.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:03 smithi156 ceph-mon[35919]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T16:11:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:03 smithi150 ceph-mon[31946]: pgmap v342: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:11:04.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:03 smithi150 ceph-mon[31946]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T16:11:04.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:03 smithi191 ceph-mon[36784]: pgmap v342: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 181 op/s 2024-03-18T16:11:04.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:03 smithi191 ceph-mon[36784]: osdmap e100: 12 total, 12 up, 12 in 2024-03-18T16:11:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 -- 172.21.15.150:0/3936027378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44068e60 msgr2=0x7f4f440692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 --2- 172.21.15.150:0/3936027378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44068e60 0x7f4f440692c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009900 tx=0x7f4f2c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 -- 172.21.15.150:0/3936027378 shutdown_connections 2024-03-18T16:11:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 --2- 172.21.15.150:0/3936027378 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 0x7f4f4410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 --2- 172.21.15.150:0/3936027378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44068e60 0x7f4f440692c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 --2- 172.21.15.150:0/3936027378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44107850 0x7f4f44107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 -- 172.21.15.150:0/3936027378 >> 172.21.15.150:0/3936027378 conn(0x7f4f44077530 msgr2=0x7f4f44077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 -- 172.21.15.150:0/3936027378 shutdown_connections 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.232+0000 7f4f4907e640 1 -- 172.21.15.150:0/3936027378 wait complete. 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 Processor -- start 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 -- start start 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 0x7f4f44197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 0x7f4f441984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f44111b70 con 0x7f4f44107850 2024-03-18T16:11:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441119f0 con 0x7f4f44068e60 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.233+0000 7f4f4907e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4f44111cf0 con 0x7f4f44069800 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 0x7f4f44197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50922/0 (socket says 172.21.15.150:50922) 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 -- 172.21.15.150:0/166041849 learned_addr learned my addr 172.21.15.150:0/166041849 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 0x7f4f441984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 -- 172.21.15.150:0/166041849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 msgr2=0x7f4f441984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 0x7f4f441984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 -- 172.21.15.150:0/166041849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 msgr2=0x7f4f44197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 0x7f4f44197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 -- 172.21.15.150:0/166041849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4419cf30 con 0x7f4f44107850 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 0x7f4f44197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4359e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00ea60 tx=0x7f4f3c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f40d99640 1 -- 172.21.15.150:0/166041849 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00ce40 con 0x7f4f44107850 2024-03-18T16:11:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f4419d220 con 0x7f4f44107850 2024-03-18T16:11:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f40d99640 1 -- 172.21.15.150:0/166041849 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3c005020 con 0x7f4f44107850 2024-03-18T16:11:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f40d99640 1 -- 172.21.15.150:0/166041849 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c010690 con 0x7f4f44107850 2024-03-18T16:11:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.234+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f441a4ab0 con 0x7f4f44107850 2024-03-18T16:11:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f4bc68740 1 Processor -- start 2024-03-18T16:11:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f40d99640 1 -- 172.21.15.150:0/166041849 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f4f3c0051d0 con 0x7f4f44107850 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f4bc68740 1 -- start start 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7130 0x55e5001e7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.237+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e5001f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e50014ba40 con 0x55e5001f0820 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e5000f4920 con 0x55e5001e7a50 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e500147ae0 con 0x55e5001e7130 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e5001f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:59580/0 (socket says 172.21.15.150:59580) 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 -- 172.21.15.150:0/989777160 learned_addr learned my addr 172.21.15.150:0/989777160 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7130 0x55e5001e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f40d99640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 0x7f4f300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 -- 172.21.15.150:0/989777160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7130 msgr2=0x55e5001e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7130 0x55e5001e7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 -- 172.21.15.150:0/989777160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 msgr2=0x55e5001f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e5001f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 -- 172.21.15.150:0/989777160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e50010c460 con 0x55e5001e7a50 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f4359e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 0x7f4f300a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e5001f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f40d99640 1 -- 172.21.15.150:0/166041849 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f4f3c0cf4a0 con 0x7f4f44107850 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.238+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f34011540 tx=0x7f4f34011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e165d518437e9a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f397fa640 1 -- 172.21.15.150:0/989777160 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400f640 con 0x55e5001e7a50 2024-03-18T16:11:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f4359e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 0x7f4f300a4ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0ccc70 tx=0x7f4f3c04b040 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f397fa640 1 -- 172.21.15.150:0/989777160 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3400fcc0 con 0x55e5001e7a50 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f397fa640 1 -- 172.21.15.150:0/989777160 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34019d30 con 0x55e5001e7a50 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f4bc68740 1 -- 172.21.15.150:0/989777160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 msgr2=0x55e5001f02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.239+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f34011540 tx=0x7f4f34011a00 comp rx=0 tx=0).stop 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 -- 172.21.15.150:0/989777160 shutdown_connections 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e5001f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7a50 0x55e5001f02e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/989777160 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7130 0x55e5001e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 -- 172.21.15.150:0/989777160 >> 172.21.15.150:0/989777160 conn(0x55e5001e1a50 msgr2=0x55e5001e1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 -- 172.21.15.150:0/989777160 shutdown_connections 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 -- 172.21.15.150:0/989777160 wait complete. 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 -- start start 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 Processor -- start 2024-03-18T16:11:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001ede00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.240+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001f0820 0x55e5001ee340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55e5001f3450 con 0x55e5001e7130 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e500107d70 con 0x55e5001f0820 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55e5001f35d0 con 0x55e5001e7a50 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f4bc68740 10 client.0 _ll_register_callbacks cb 0x55e4ffe5fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50930/0 (socket says 172.21.15.150:50930) 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f3bfff640 1 -- 172.21.15.150:0/1263924893 learned_addr learned my addr 172.21.15.150:0/1263924893 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001f0820 0x55e5001ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 -- 172.21.15.150:0/1263924893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001f0820 msgr2=0x55e5001ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001f0820 0x55e5001ee340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 -- 172.21.15.150:0/1263924893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 msgr2=0x55e5001ed8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 -- 172.21.15.150:0/1263924893 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4ffe60210 con 0x55e5001e7a50 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001ede00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f34000c00 tx=0x7f4f34005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b914f6acba4ca00c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34019410 con 0x55e5001e7a50 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f4f3400f850 con 0x55e5001e7a50 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.241+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f3400fcc0 con 0x55e5001e7a50 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4f34030050 con 0x55e5001e7a50 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 10 client.26143 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e4fff2c620 con 0x55e5001e7a50 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3400fcc0 con 0x55e5001e7a50 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4f340046c0 con 0x55e5001e7a50 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 10 client.26143 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 10 client.26143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 20 client.26143 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5002008f0 0x55e500202ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 4 client.26143 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e500158f80 con 0x55e5002008f0 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.242+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5002008f0 0x55e500202ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.243+0000 7f4f2a7fc640 10 client.26143 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.243+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5002008f0 0x55e500202ce0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=954a84994967458d server_cookie=20dd43d939ae4cbc in_seq=0 out_seq=0 2024-03-18T16:11:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/1263924893 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e500158f80 con 0x55e5002008f0 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f2a7fc640 10 client.26143 handle_command_reply: tid=0 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 client.26143 shutdown 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 msgr2=0x55e5001ede00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001ede00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f34000c00 tx=0x7f4f34005d30 comp rx=0 tx=0).stop 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 shutdown_connections 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e5002008f0 0x55e500202ce0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001f0820 0x55e5001ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001ede00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/1263924893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7130 0x55e5001ed8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f2a7fc640 0 client.26143 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 >> 172.21.15.150:0/1263924893 conn(0x55e5001e1a50 msgr2=0x55e5001e37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 shutdown_connections 2024-03-18T16:11:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 1 -- 172.21.15.150:0/1263924893 wait complete. 2024-03-18T16:11:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.244+0000 7f4f4bc68740 20 client.26143 trim_cache size 0 max 16384 2024-03-18T16:11:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 Processor -- start 2024-03-18T16:11:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 -- start start 2024-03-18T16:11:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001efff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441afde0 con 0x55e5001f0820 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441ad6b0 con 0x55e5001e7130 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441ab520 con 0x55e5001e7a50 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:45020/0 (socket says 172.21.15.150:45020) 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50946/0 (socket says 172.21.15.150:50946) 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f42d9d640 1 -- 172.21.15.150:0/965368368 learned_addr learned my addr 172.21.15.150:0/965368368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 -- 172.21.15.150:0/965368368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 msgr2=0x55e5001fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.266 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.259+0000 7f4f3bfff640 1 -- 172.21.15.150:0/965368368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 msgr2=0x55e5001efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f3bfff640 1 -- 172.21.15.150:0/965368368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5001fc730 con 0x55e5001f0820 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x55e500203340 tx=0x7f4f2c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebad6c45de8f310 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f38ff9640 1 -- 172.21.15.150:0/965368368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c04f680 con 0x55e5001f0820 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f38ff9640 1 -- 172.21.15.150:0/965368368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c004ba0 con 0x55e5001f0820 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.260+0000 7f4f38ff9640 1 -- 172.21.15.150:0/965368368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0054f0 con 0x55e5001f0820 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- 172.21.15.150:0/965368368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 msgr2=0x55e500208400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x55e500203340 tx=0x7f4f2c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- 172.21.15.150:0/965368368 shutdown_connections 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001f0820 0x55e500208400 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001e7a50 0x55e5001fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/965368368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- 172.21.15.150:0/965368368 >> 172.21.15.150:0/965368368 conn(0x55e5001e1a50 msgr2=0x55e5001ec190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- 172.21.15.150:0/965368368 shutdown_connections 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- 172.21.15.150:0/965368368 wait complete. 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- start start 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 Processor -- start 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001f0820 0x55e50025fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441abb70 con 0x55e5001e7a50 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441acfe0 con 0x55e5001e7130 2024-03-18T16:11:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441ac240 con 0x55e5001f0820 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.261+0000 7f4f4bc68740 10 client.0 _ll_register_callbacks cb 0x7f4f3c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:59610/0 (socket says 172.21.15.150:59610) 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f42d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50950/0 (socket says 172.21.15.150:50950) 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 -- 172.21.15.150:0/4212303766 learned_addr learned my addr 172.21.15.150:0/4212303766 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001f0820 0x55e50025fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 -- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001f0820 msgr2=0x55e50025fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001f0820 0x55e50025fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 -- 172.21.15.150:0/4212303766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 msgr2=0x55e5001f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 -- 172.21.15.150:0/4212303766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5002600b0 con 0x55e5001e7130 2024-03-18T16:11:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f3bfff640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009900 tx=0x7f4f2c0050d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d54e4fdabb8eda2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c04f480 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f2c004ba0 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8571+0+0 (secure 0 0 0) 0x7f4f2c04f620 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c04f820 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f4bc68740 10 client.26090 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e500267470 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.262+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c004ba0 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4f2c066440 con 0x55e5001e7130 2024-03-18T16:11:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 10 client.26090 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 10 client.26090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 20 client.26090 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e500267930 0x55e500269d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 4 client.26090 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5001532d0 con 0x55e500267930 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e500267930 0x55e500269d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f2a7fc640 10 client.26090 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.263+0000 7f4f42d9d640 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e500267930 0x55e500269d20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e84b1f159958aa server_cookie=ded0175ee41fd380 in_seq=0 out_seq=0 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.276+0000 7f4f2a7fc640 1 -- 172.21.15.150:0/4212303766 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5001532d0 con 0x55e500267930 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.276+0000 7f4f2a7fc640 10 client.26090 handle_command_reply: tid=0 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.276+0000 7f4f4bc68740 1 client.26090 shutdown 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 msgr2=0x55e5001f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009900 tx=0x7f4f2c0050d0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 shutdown_connections 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55e500267930 0x55e500269d20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55e5001f0820 0x55e50025fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f2a7fc640 0 client.26090 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e5001e7a50 0x55e5001f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 --2- 172.21.15.150:0/4212303766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5001e7130 0x55e5001f49b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 >> 172.21.15.150:0/4212303766 conn(0x55e5001e1a50 msgr2=0x55e5001ed860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 shutdown_connections 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 1 -- 172.21.15.150:0/4212303766 wait complete. 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4bc68740 20 client.26090 trim_cache size 0 max 16384 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:11:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:11:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 msgr2=0x7f4f300a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 0x7f4f300a4ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0ccc70 tx=0x7f4f3c04b040 comp rx=0 tx=0).stop 2024-03-18T16:11:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 msgr2=0x7f4f4419c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.277+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00ea60 tx=0x7f4f3c00ef20 comp rx=0 tx=0).stop 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 shutdown_connections 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f4f300a2a10 0x7f4f300a4ed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f44107850 0x7f4f4419c800 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f44069800 0x7f4f441984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 --2- 172.21.15.150:0/166041849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f44068e60 0x7f4f44197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 >> 172.21.15.150:0/166041849 conn(0x7f4f44077530 msgr2=0x7f4f440ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 shutdown_connections 2024-03-18T16:11:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:04.278+0000 7f4f4907e640 1 -- 172.21.15.150:0/166041849 wait complete. 2024-03-18T16:11:04.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1120789 10.20 MB/sec execute 469 sec latency 21.218 ms 2024-03-18T16:11:05.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1122746 10.21 MB/sec execute 470 sec latency 58.886 ms 2024-03-18T16:11:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:05 smithi156 ceph-mon[35919]: pgmap v344: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-03-18T16:11:06.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:05 smithi150 ceph-mon[31946]: pgmap v344: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-03-18T16:11:06.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:05 smithi191 ceph-mon[36784]: pgmap v344: 97 pgs: 97 active+clean; 646 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 217 op/s 2024-03-18T16:11:06.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1124121 10.20 MB/sec execute 471 sec latency 37.429 ms 2024-03-18T16:11:07.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1126242 10.21 MB/sec execute 472 sec latency 20.926 ms 2024-03-18T16:11:08.014 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:07 smithi191 ceph-mon[36784]: pgmap v345: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:07 smithi156 ceph-mon[35919]: pgmap v345: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:07 smithi150 ceph-mon[31946]: pgmap v345: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:08.342 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1127437 10.20 MB/sec execute 473 sec latency 37.532 ms 2024-03-18T16:11:09.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1128716 10.19 MB/sec execute 474 sec latency 20.587 ms 2024-03-18T16:11:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:09 smithi156 ceph-mon[35919]: pgmap v346: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:09 smithi156 ceph-mon[35919]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T16:11:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:09 smithi150 ceph-mon[31946]: pgmap v346: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:10.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:09 smithi150 ceph-mon[31946]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T16:11:10.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:09 smithi191 ceph-mon[36784]: pgmap v346: 97 pgs: 97 active+clean; 605 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 241 op/s 2024-03-18T16:11:10.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:09 smithi191 ceph-mon[36784]: osdmap e101: 12 total, 12 up, 12 in 2024-03-18T16:11:10.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1130173 10.18 MB/sec execute 475 sec latency 48.150 ms 2024-03-18T16:11:11.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1131992 10.18 MB/sec execute 476 sec latency 50.362 ms 2024-03-18T16:11:12.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:11 smithi156 ceph-mon[35919]: pgmap v348: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 291 op/s 2024-03-18T16:11:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:11 smithi150 ceph-mon[31946]: pgmap v348: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 291 op/s 2024-03-18T16:11:12.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:11 smithi191 ceph-mon[36784]: pgmap v348: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 291 op/s 2024-03-18T16:11:12.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1133777 10.18 MB/sec execute 477 sec latency 24.025 ms 2024-03-18T16:11:13.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1135518 10.17 MB/sec execute 478 sec latency 82.851 ms 2024-03-18T16:11:14.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:13 smithi156 ceph-mon[35919]: pgmap v349: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 235 op/s 2024-03-18T16:11:14.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:13 smithi150 ceph-mon[31946]: pgmap v349: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 235 op/s 2024-03-18T16:11:14.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:13 smithi191 ceph-mon[36784]: pgmap v349: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 235 op/s 2024-03-18T16:11:14.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1137273 10.18 MB/sec execute 479 sec latency 25.993 ms 2024-03-18T16:11:15.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1139071 10.17 MB/sec execute 480 sec latency 27.478 ms 2024-03-18T16:11:16.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:15 smithi156 ceph-mon[35919]: pgmap v350: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-18T16:11:16.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:15 smithi150 ceph-mon[31946]: pgmap v350: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-18T16:11:16.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:15 smithi191 ceph-mon[36784]: pgmap v350: 97 pgs: 97 active+clean; 618 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-18T16:11:16.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1140663 10.17 MB/sec execute 481 sec latency 70.545 ms 2024-03-18T16:11:17.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1142405 10.17 MB/sec execute 482 sec latency 23.128 ms 2024-03-18T16:11:17.369 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- 172.21.15.150:0/1533462945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c1012f0 msgr2=0x7f075c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 --2- 172.21.15.150:0/1533462945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c1012f0 0x7f075c10d510 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0754009ca0 tx=0x7f075402c0f0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- 172.21.15.150:0/1533462945 shutdown_connections 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 --2- 172.21.15.150:0/1533462945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c1012f0 0x7f075c10d510 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 --2- 172.21.15.150:0/1533462945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c100930 0x7f075c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 --2- 172.21.15.150:0/1533462945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c105680 0x7f075c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- 172.21.15.150:0/1533462945 >> 172.21.15.150:0/1533462945 conn(0x7f075c077530 msgr2=0x7f075c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- 172.21.15.150:0/1533462945 shutdown_connections 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- 172.21.15.150:0/1533462945 wait complete. 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 Processor -- start 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.366+0000 7f0764219640 1 -- start start 2024-03-18T16:11:17.370 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c100930 0x7f075c196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c1012f0 0x7f075c1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f075c110ba0 con 0x7f075c105680 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f075c110a20 con 0x7f075c1012f0 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f075c110d20 con 0x7f075c100930 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42286/0 (socket says 172.21.15.150:42286) 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 -- 172.21.15.150:0/781799059 learned_addr learned my addr 172.21.15.150:0/781799059 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0761fb5640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c100930 0x7f075c196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07617b4640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c1012f0 0x7f075c1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 -- 172.21.15.150:0/781799059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c100930 msgr2=0x7f075c196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c100930 0x7f075c196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 -- 172.21.15.150:0/781799059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c1012f0 msgr2=0x7f075c1974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c1012f0 0x7f075c1974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 -- 172.21.15.150:0/781799059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c19bf70 con 0x7f075c105680 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f07627b6640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f075402c700 tx=0x7f0754032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0752ffd640 1 -- 172.21.15.150:0/781799059 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754042020 con 0x7f075c105680 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075c19c200 con 0x7f075c105680 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0752ffd640 1 -- 172.21.15.150:0/781799059 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0754034640 con 0x7f075c105680 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0752ffd640 1 -- 172.21.15.150:0/781799059 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754039040 con 0x7f075c105680 2024-03-18T16:11:17.372 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.367+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075c1a3af0 con 0x7f075c105680 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.369+0000 7f0752ffd640 1 -- 172.21.15.150:0/781799059 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f0754045020 con 0x7f075c105680 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.370+0000 7f0752ffd640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 0x7f07480a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.370+0000 7f0761fb5640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 0x7f07480a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.370+0000 7f0752ffd640 1 -- 172.21.15.150:0/781799059 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f07540f0bc0 con 0x7f075c105680 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.370+0000 7f0766e0a740 1 Processor -- start 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.370+0000 7f0766e0a740 1 -- start start 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7781510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f778a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.377 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5588f76e5a40 con 0x5588f778a820 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588f768e920 con 0x5588f7781130 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0766e0a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5588f76e1ae0 con 0x5588f7781a50 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0761fb5640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 0x7f07480a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f074c0095f0 tx=0x7f074c005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42296/0 (socket says 172.21.15.150:42296) 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 -- 172.21.15.150:0/2786151862 learned_addr learned my addr 172.21.15.150:0/2786151862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07617b4640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f778a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0761fb5640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7781510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 -- 172.21.15.150:0/2786151862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 msgr2=0x5588f778a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f778a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 -- 172.21.15.150:0/2786151862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 msgr2=0x5588f7781510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7781510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.378 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 -- 172.21.15.150:0/2786151862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f76a6460 con 0x5588f778a820 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0761fb5640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7781510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f07627b6640 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f07540c55a0 tx=0x7f07540bd270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f796b199c9625 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0747fff640 1 -- 172.21.15.150:0/2786151862 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540c1480 con 0x5588f778a820 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0747fff640 1 -- 172.21.15.150:0/2786151862 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07540c5ad0 con 0x5588f778a820 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.371+0000 7f0747fff640 1 -- 172.21.15.150:0/2786151862 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540b9a50 con 0x5588f778a820 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 -- 172.21.15.150:0/2786151862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 msgr2=0x5588f778cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f07540c55a0 tx=0x7f07540bd270 comp rx=0 tx=0).stop 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 -- 172.21.15.150:0/2786151862 shutdown_connections 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778cc10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f778a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2786151862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7781510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 -- 172.21.15.150:0/2786151862 >> 172.21.15.150:0/2786151862 conn(0x5588f777ba50 msgr2=0x5588f777be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 -- 172.21.15.150:0/2786151862 shutdown_connections 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.372+0000 7f0766e0a740 1 -- 172.21.15.150:0/2786151862 wait complete. 2024-03-18T16:11:17.379 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 -- start start 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 Processor -- start 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 0x5588f7787950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781a50 0x5588f7787e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5588f778d210 con 0x5588f778a820 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588f76a1d70 con 0x5588f7781a50 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x5588f778d390 con 0x5588f7781130 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0766e0a740 10 client.0 _ll_register_callbacks cb 0x5588f73f9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0761fb5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 0x5588f7787950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42312/0 (socket says 172.21.15.150:42312) 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 -- 172.21.15.150:0/2274492731 learned_addr learned my addr 172.21.15.150:0/2274492731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07617b4640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781a50 0x5588f7787e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 -- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 msgr2=0x5588f7787950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.380 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 0x5588f7787950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 -- 172.21.15.150:0/2274492731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781a50 msgr2=0x5588f7787e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781a50 0x5588f7787e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f07627b6640 1 -- 172.21.15.150:0/2274492731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f73fa210 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.373+0000 7f0761fb5640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 0x5588f7787950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07627b6640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f07540b9840 tx=0x7f07540ee440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd08d13ac7a7367 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540c1480 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f07540c1120 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07540c5ad0 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0754003970 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 10 client.16338 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5588f74c6620 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07540c5ad0 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f07540c5640 con 0x5588f778a820 2024-03-18T16:11:17.381 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 10 client.16338 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 20 client.16338 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f779a670 0x5588f779ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 4 client.16338 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.374+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f76f2f80 con 0x5588f779a670 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.375+0000 7f0761fb5640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f779a670 0x5588f779ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.375+0000 7f07457fa640 10 client.16338 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:17.382 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.375+0000 7f0761fb5640 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f779a670 0x5588f779ca60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5491b634025ca460 server_cookie=4ef2cfb09278b219 in_seq=0 out_seq=0 2024-03-18T16:11:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f07457fa640 1 -- 172.21.15.150:0/2274492731 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588f76f2f80 con 0x5588f779a670 2024-03-18T16:11:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f07457fa640 10 client.16338 handle_command_reply: tid=0 2024-03-18T16:11:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 client.16338 shutdown 2024-03-18T16:11:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 msgr2=0x5588f77883d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.383 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f07540b9840 tx=0x7f07540ee440 comp rx=0 tx=0).stop 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 shutdown_connections 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f779a670 0x5588f779ca60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f77883d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781a50 0x5588f7787e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 --2- 172.21.15.150:0/2274492731 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781130 0x5588f7787950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 >> 172.21.15.150:0/2274492731 conn(0x5588f777ba50 msgr2=0x5588f777d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f07457fa640 0 client.16338 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 shutdown_connections 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.376+0000 7f0766e0a740 1 -- 172.21.15.150:0/2274492731 wait complete. 2024-03-18T16:11:17.384 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.377+0000 7f0766e0a740 20 client.16338 trim_cache size 0 max 16384 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 Processor -- start 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 -- start start 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7795f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f7786ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1aeb60 con 0x5588f778a820 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1ac430 con 0x5588f7781130 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0766e0a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1aa2a0 con 0x5588f7781a50 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42328/0 (socket says 172.21.15.150:42328) 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f07627b6640 1 -- 172.21.15.150:0/604436803 learned_addr learned my addr 172.21.15.150:0/604436803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f0761fb5640 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7795f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.391+0000 7f07617b4640 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f7786ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 -- 172.21.15.150:0/604436803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 msgr2=0x5588f7786ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f7786ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 -- 172.21.15.150:0/604436803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 msgr2=0x5588f7795f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7795f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 -- 172.21.15.150:0/604436803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f777b580 con 0x5588f778a820 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07627b6640 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f07540bceb0 tx=0x7f075402df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15cb23b3402792c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07477fe640 1 -- 172.21.15.150:0/604436803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540c5ad0 con 0x5588f778a820 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07477fe640 1 -- 172.21.15.150:0/604436803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07540c1480 con 0x5588f778a820 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.392+0000 7f07477fe640 1 -- 172.21.15.150:0/604436803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540dd0f0 con 0x5588f778a820 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- 172.21.15.150:0/604436803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 msgr2=0x5588f777aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f07540bceb0 tx=0x7f075402df90 comp rx=0 tx=0).stop 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- 172.21.15.150:0/604436803 shutdown_connections 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f777aa30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f7786ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 --2- 172.21.15.150:0/604436803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f7795f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- 172.21.15.150:0/604436803 >> 172.21.15.150:0/604436803 conn(0x5588f777ba50 msgr2=0x5588f777a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- 172.21.15.150:0/604436803 shutdown_connections 2024-03-18T16:11:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- 172.21.15.150:0/604436803 wait complete. 2024-03-18T16:11:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 -- start start 2024-03-18T16:11:17.397 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.393+0000 7f0766e0a740 1 Processor -- start 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f77fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f77fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1aa8f0 con 0x5588f778a820 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1abd60 con 0x5588f7781130 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f075c1aafc0 con 0x5588f7781a50 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 10 client.0 _ll_register_callbacks cb 0x7f07540c1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0761fb5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f77fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42334/0 (socket says 172.21.15.150:42334) 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 -- 172.21.15.150:0/3219774237 learned_addr learned my addr 172.21.15.150:0/3219774237 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07617b4640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f77fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 -- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 msgr2=0x5588f77fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f77fd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 -- 172.21.15.150:0/3219774237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 msgr2=0x5588f77fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f77fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 -- 172.21.15.150:0/3219774237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f778dd20 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0761fb5640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f77fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07627b6640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f075402c6d0 tx=0x7f07540b9050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7884adb1bb3e0f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07457fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540bcb00 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f07540b9130 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07540c1480 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07540c1270 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 10 client.16344 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.394+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5588f778dd20 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07540c1480 con 0x5588f778a820 2024-03-18T16:11:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f075409f050 con 0x5588f778a820 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 10 client.16344 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 10 client.16344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 20 client.16344 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f78047b0 0x5588f7806ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 4 client.16344 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588f76ed2d0 con 0x5588f78047b0 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0761fb5640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f78047b0 0x5588f7806ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f07457fa640 10 client.16344 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.395+0000 7f0761fb5640 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f78047b0 0x5588f7806ba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54b2c62df7444c5 server_cookie=f39ee1ce397dbd39 in_seq=0 out_seq=0 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f07457fa640 1 -- 172.21.15.150:0/3219774237 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588f76ed2d0 con 0x5588f78047b0 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f07457fa640 10 client.16344 handle_command_reply: tid=0 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 client.16344 shutdown 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 msgr2=0x5588f778d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f075402c6d0 tx=0x7f07540b9050 comp rx=0 tx=0).stop 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 shutdown_connections 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x5588f78047b0 0x5588f7806ba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5588f778a820 0x5588f778d630 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f07457fa640 0 client.16344 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5588f7781a50 0x5588f77fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 --2- 172.21.15.150:0/3219774237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588f7781130 0x5588f77fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 >> 172.21.15.150:0/3219774237 conn(0x5588f777ba50 msgr2=0x5588f7789890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 shutdown_connections 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 1 -- 172.21.15.150:0/3219774237 wait complete. 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.408+0000 7f0766e0a740 20 client.16344 trim_cache size 0 max 16384 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:11:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:11:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 msgr2=0x7f07480a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 0x7f07480a4ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f074c0095f0 tx=0x7f074c005f90 comp rx=0 tx=0).stop 2024-03-18T16:11:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 msgr2=0x7f075c19b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f075402c700 tx=0x7f0754032f20 comp rx=0 tx=0).stop 2024-03-18T16:11:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 shutdown_connections 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f07480a2a10 0x7f07480a4ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f075c105680 0x7f075c19b840 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f075c1012f0 0x7f075c1974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 --2- 172.21.15.150:0/781799059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f075c100930 0x7f075c196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 >> 172.21.15.150:0/781799059 conn(0x7f075c077530 msgr2=0x7f075c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 shutdown_connections 2024-03-18T16:11:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:17.409+0000 7f0764219640 1 -- 172.21.15.150:0/781799059 wait complete. 2024-03-18T16:11:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:17 smithi156 ceph-mon[35919]: pgmap v351: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:17 smithi150 ceph-mon[31946]: pgmap v351: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:18.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:17 smithi191 ceph-mon[36784]: pgmap v351: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:18.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1144210 10.17 MB/sec execute 483 sec latency 52.460 ms 2024-03-18T16:11:19.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1145636 10.16 MB/sec execute 484 sec latency 37.152 ms 2024-03-18T16:11:20.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:19 smithi156 ceph-mon[35919]: pgmap v352: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:19 smithi150 ceph-mon[31946]: pgmap v352: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:20.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:19 smithi191 ceph-mon[36784]: pgmap v352: 97 pgs: 97 active+clean; 628 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-18T16:11:20.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1147731 10.17 MB/sec execute 485 sec latency 17.398 ms 2024-03-18T16:11:21.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1148984 10.16 MB/sec execute 486 sec latency 58.071 ms 2024-03-18T16:11:22.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:21 smithi156 ceph-mon[35919]: pgmap v353: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.5 MiB/s wr, 283 op/s 2024-03-18T16:11:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:21 smithi150 ceph-mon[31946]: pgmap v353: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.5 MiB/s wr, 283 op/s 2024-03-18T16:11:22.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:21 smithi191 ceph-mon[36784]: pgmap v353: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.5 MiB/s wr, 283 op/s 2024-03-18T16:11:22.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1151223 10.17 MB/sec execute 487 sec latency 18.713 ms 2024-03-18T16:11:23.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:22 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:23.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:22 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:23.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:22 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:23.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1152444 10.16 MB/sec execute 488 sec latency 45.446 ms 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: pgmap v354: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: pgmap v354: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:24.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:24.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:24.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: pgmap v354: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:24.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:24.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:24.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:11:24.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1154620 10.16 MB/sec execute 489 sec latency 20.065 ms 2024-03-18T16:11:25.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:24 smithi156 ceph-mon[35919]: pgmap v355: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:25.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:24 smithi150 ceph-mon[31946]: pgmap v355: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:25.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:24 smithi191 ceph-mon[36784]: pgmap v355: 97 pgs: 97 active+clean; 639 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 181 op/s 2024-03-18T16:11:25.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1155978 10.15 MB/sec execute 490 sec latency 19.174 ms 2024-03-18T16:11:26.346 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1157924 10.16 MB/sec execute 491 sec latency 76.784 ms 2024-03-18T16:11:27.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1159110 10.15 MB/sec execute 492 sec latency 21.108 ms 2024-03-18T16:11:27.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:27 smithi191 ceph-mon[36784]: pgmap v356: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-18T16:11:28.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:27 smithi156 ceph-mon[35919]: pgmap v356: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-18T16:11:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:27 smithi150 ceph-mon[31946]: pgmap v356: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.6 MiB/s wr, 260 op/s 2024-03-18T16:11:28.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1161114 10.15 MB/sec execute 493 sec latency 22.829 ms 2024-03-18T16:11:29.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1162617 10.14 MB/sec execute 494 sec latency 67.158 ms 2024-03-18T16:11:29.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:29 smithi191 ceph-mon[36784]: pgmap v357: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-18T16:11:30.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:29 smithi156 ceph-mon[35919]: pgmap v357: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-18T16:11:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:29 smithi150 ceph-mon[31946]: pgmap v357: 97 pgs: 97 active+clean; 644 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-18T16:11:30.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1164685 10.15 MB/sec execute 495 sec latency 20.040 ms 2024-03-18T16:11:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 -- 172.21.15.150:0/2082396575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0105680 msgr2=0x7f84c0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 --2- 172.21.15.150:0/2082396575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0105680 0x7f84c0105a60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009900 tx=0x7f84b002bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 -- 172.21.15.150:0/2082396575 shutdown_connections 2024-03-18T16:11:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 --2- 172.21.15.150:0/2082396575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c01012f0 0x7f84c010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.501 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 --2- 172.21.15.150:0/2082396575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c0100930 0x7f84c0100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.502 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 --2- 172.21.15.150:0/2082396575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0105680 0x7f84c0105a60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.502 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 -- 172.21.15.150:0/2082396575 >> 172.21.15.150:0/2082396575 conn(0x7f84c0077530 msgr2=0x7f84c0077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.502 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 -- 172.21.15.150:0/2082396575 shutdown_connections 2024-03-18T16:11:30.502 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.499+0000 7f84c60f7640 1 -- 172.21.15.150:0/2082396575 wait complete. 2024-03-18T16:11:30.502 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 Processor -- start 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 -- start start 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c01012f0 0x7f84c0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 0x7f84c01a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0110ba0 con 0x7f84c0100930 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0110a20 con 0x7f84c01012f0 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84c60f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0110d20 con 0x7f84c0105680 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84beffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c01012f0 0x7f84c0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 0x7f84c01a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42246/0 (socket says 172.21.15.150:42246) 2024-03-18T16:11:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 -- 172.21.15.150:0/1458249052 learned_addr learned my addr 172.21.15.150:0/1458249052 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 -- 172.21.15.150:0/1458249052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 msgr2=0x7f84c01a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 0x7f84c01a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 -- 172.21.15.150:0/1458249052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c01012f0 msgr2=0x7f84c0199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c01012f0 0x7f84c0199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bf7fe640 1 -- 172.21.15.150:0/1458249052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c01a44a0 con 0x7f84c0100930 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.500+0000 7f84bffff640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 0x7f84c01a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005940 tx=0x7f84b0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84bcff9640 1 -- 172.21.15.150:0/1458249052 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0041070 con 0x7f84c0100930 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01a4670 con 0x7f84c0100930 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c01a4ad0 con 0x7f84c0100930 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84bcff9640 1 -- 172.21.15.150:0/1458249052 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b0004c20 con 0x7f84c0100930 2024-03-18T16:11:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.501+0000 7f84bcff9640 1 -- 172.21.15.150:0/1458249052 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b004b060 con 0x7f84c0100930 2024-03-18T16:11:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.503+0000 7f84bcff9640 1 -- 172.21.15.150:0/1458249052 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f84b0044070 con 0x7f84c0100930 2024-03-18T16:11:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.503+0000 7f84c8db7740 1 Processor -- start 2024-03-18T16:11:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.503+0000 7f84c8db7740 1 -- start start 2024-03-18T16:11:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754f130 0x557b1754f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 0x557b175582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557b174b3a40 con 0x557b1754f130 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557b1745c920 con 0x557b17558820 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84c8db7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557b174afae0 con 0x557b1754fa50 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754f130 0x557b1754f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bcff9640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 0x7f84a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 0x557b175582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:48510/0 (socket says 172.21.15.150:48510) 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 0x557b175582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:42546/0 (socket says 172.21.15.150:42546) 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 -- 172.21.15.150:0/1714678945 learned_addr learned my addr 172.21.15.150:0/1714678945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bffff640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 0x7f84a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 -- 172.21.15.150:0/1714678945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 msgr2=0x557b175582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 0x557b175582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 -- 172.21.15.150:0/1714678945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754f130 msgr2=0x557b1754f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754f130 0x557b1754f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 -- 172.21.15.150:0/1714678945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b17474460 con 0x557b17558820 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bcff9640 1 -- 172.21.15.150:0/1458249052 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f84b00f06e0 con 0x7f84c0100930 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84beffd640 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009c90 tx=0x7f84b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb5903ff54f4142 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84bffff640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 0x7f84a80a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009930 tx=0x7f84b8004ab0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.504+0000 7f84a5ffb640 1 -- 172.21.15.150:0/1714678945 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b401e070 con 0x557b17558820 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84a5ffb640 1 -- 172.21.15.150:0/1714678945 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b4019b10 con 0x557b17558820 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84a5ffb640 1 -- 172.21.15.150:0/1714678945 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b400ea50 con 0x557b17558820 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 -- 172.21.15.150:0/1714678945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 msgr2=0x557b1755ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009c90 tx=0x7f84b4011810 comp rx=0 tx=0).stop 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 -- 172.21.15.150:0/1714678945 shutdown_connections 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755ac10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754fa50 0x557b175582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 --2- 172.21.15.150:0/1714678945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754f130 0x557b1754f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 -- 172.21.15.150:0/1714678945 >> 172.21.15.150:0/1714678945 conn(0x557b17549a50 msgr2=0x557b17549e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 -- 172.21.15.150:0/1714678945 shutdown_connections 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.505+0000 7f84c8db7740 1 -- 172.21.15.150:0/1714678945 wait complete. 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 -- start start 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 Processor -- start 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17555770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557b1755b450 con 0x557b1754fa50 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x557b1746fd70 con 0x557b17558820 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x557b1755b5d0 con 0x557b1754f130 2024-03-18T16:11:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84c8db7740 10 client.0 _ll_register_callbacks cb 0x557b171c7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42254/0 (socket says 172.21.15.150:42254) 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84bf7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17555770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84bffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 -- 172.21.15.150:0/3024789571 learned_addr learned my addr 172.21.15.150:0/3024789571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 -- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 msgr2=0x557b17555770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17555770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 -- 172.21.15.150:0/3024789571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 msgr2=0x557b175561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84beffd640 1 -- 172.21.15.150:0/3024789571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b171c8210 con 0x557b1754fa50 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.506+0000 7f84bffff640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84beffd640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f84b40199b0 tx=0x7f84b400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d00f8bd419755e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b402a060 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f84b4004030 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84b4019b10 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84b401e040 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84c8db7740 10 client.16356 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b17294620 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b4019b10 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f84b400e750 con 0x557b1754fa50 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84c8db7740 10 client.16356 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.507+0000 7f84c8db7740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84c8db7740 20 client.16356 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b17568820 0x557b1756ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84c8db7740 4 client.16356 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b174c0f80 con 0x557b17568820 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b17568820 0x557b1756ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f849effd640 10 client.16356 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.508+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b17568820 0x557b1756ac10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=972f6cef7876725a server_cookie=557c41b5cb8c7ee5 in_seq=0 out_seq=0 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f849effd640 1 -- 172.21.15.150:0/3024789571 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b174c0f80 con 0x557b17568820 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f849effd640 10 client.16356 handle_command_reply: tid=0 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 client.16356 shutdown 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 msgr2=0x557b17555cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f84b40199b0 tx=0x7f84b400e530 comp rx=0 tx=0).stop 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 shutdown_connections 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b17568820 0x557b1756ac10 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17555cb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f849effd640 0 client.16356 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3024789571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17555770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 >> 172.21.15.150:0/3024789571 conn(0x557b17549a50 msgr2=0x557b1754b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 shutdown_connections 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 1 -- 172.21.15.150:0/3024789571 wait complete. 2024-03-18T16:11:30.518 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.509+0000 7f84c8db7740 20 client.16356 trim_cache size 0 max 16384 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.525+0000 7f84c8db7740 1 Processor -- start 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.525+0000 7f84c8db7740 1 -- start start 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17557ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175548d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac010ff0 con 0x557b1754fa50 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00e8c0 con 0x557b17558820 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84c8db7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00c730 con 0x557b1754f130 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42266/0 (socket says 172.21.15.150:42266) 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 -- 172.21.15.150:0/2315833659 learned_addr learned my addr 172.21.15.150:0/2315833659 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17557ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84bffff640 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 -- 172.21.15.150:0/2315833659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 msgr2=0x557b17557ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17557ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 -- 172.21.15.150:0/2315833659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 msgr2=0x557b175548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 -- 172.21.15.150:0/2315833659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b17549880 con 0x557b1754fa50 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84beffd640 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f84b4011850 tx=0x7f84b400e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec334a3a85f3fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84a57fa640 1 -- 172.21.15.150:0/2315833659 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b402a070 con 0x557b1754fa50 2024-03-18T16:11:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84a57fa640 1 -- 172.21.15.150:0/2315833659 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b400ed40 con 0x557b1754fa50 2024-03-18T16:11:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.526+0000 7f84a57fa640 1 -- 172.21.15.150:0/2315833659 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4005460 con 0x557b1754fa50 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- 172.21.15.150:0/2315833659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 msgr2=0x557b17564220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f84b4011850 tx=0x7f84b400e5a0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- 172.21.15.150:0/2315833659 shutdown_connections 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b175548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b17564220 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 --2- 172.21.15.150:0/2315833659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17557ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- 172.21.15.150:0/2315833659 >> 172.21.15.150:0/2315833659 conn(0x557b17549a50 msgr2=0x557b17555730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- 172.21.15.150:0/2315833659 shutdown_connections 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- 172.21.15.150:0/2315833659 wait complete. 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.527+0000 7f84c8db7740 1 -- start start 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 Processor -- start 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17560890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00cd80 con 0x557b1754fa50 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00e1f0 con 0x557b17558820 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00d450 con 0x557b1754f130 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84c8db7740 10 client.0 _ll_register_callbacks cb 0x7f84b401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42274/0 (socket says 172.21.15.150:42274) 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 -- 172.21.15.150:0/3751952803 learned_addr learned my addr 172.21.15.150:0/3751952803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17560890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84bffff640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 -- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 msgr2=0x557b17560890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17560890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 -- 172.21.15.150:0/3751952803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 msgr2=0x557b1755c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84beffd640 1 -- 172.21.15.150:0/3751952803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b1755ac10 con 0x557b1754fa50 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.528+0000 7f84bffff640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84beffd640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f84b4004600 tx=0x7f84b400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2283b125ddb80c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b403a030 con 0x557b1754fa50 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84b400ed40 con 0x557b1754fa50 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f84b4009e40 con 0x557b1754fa50 2024-03-18T16:11:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f84b4044050 con 0x557b1754fa50 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 10 client.16362 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b175d2a30 con 0x557b1754fa50 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b400ed40 con 0x557b1754fa50 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f84b40049b0 con 0x557b1754fa50 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 10 client.16362 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 20 client.16362 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b175d30e0 0x557b175d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 4 client.16362 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b174bb2d0 con 0x557b175d30e0 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.529+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b175d30e0 0x557b175d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.530+0000 7f84bf7fe640 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b175d30e0 0x557b175d5500 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9c639a0a2cddde2 server_cookie=ad9b3153191c7a42 in_seq=0 out_seq=0 2024-03-18T16:11:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.530+0000 7f849effd640 10 client.16362 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f849effd640 1 -- 172.21.15.150:0/3751952803 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b174bb2d0 con 0x557b175d30e0 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f849effd640 10 client.16362 handle_command_reply: tid=0 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 client.16362 shutdown 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 msgr2=0x557b1755bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f84b4004600 tx=0x7f84b400ebd0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 shutdown_connections 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x557b175d30e0 0x557b175d5500 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b17558820 0x557b1755c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557b1754fa50 0x557b1755bc20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f849effd640 0 client.16362 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 --2- 172.21.15.150:0/3751952803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x557b1754f130 0x557b17560890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 >> 172.21.15.150:0/3751952803 conn(0x557b17549a50 msgr2=0x557b17548bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 shutdown_connections 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 1 -- 172.21.15.150:0/3751952803 wait complete. 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.542+0000 7f84c8db7740 20 client.16362 trim_cache size 0 max 16384 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:11:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:11:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 msgr2=0x7f84a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 0x7f84a80a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009930 tx=0x7f84b8004ab0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 msgr2=0x7f84c0198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f84b0005940 tx=0x7f84b0004270 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 shutdown_connections 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f84a80a2a10 0x7f84a80a4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84c0105680 0x7f84c01a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84c01012f0 0x7f84c0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 --2- 172.21.15.150:0/1458249052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c0100930 0x7f84c0198ee0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 >> 172.21.15.150:0/1458249052 conn(0x7f84c0077530 msgr2=0x7f84c00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 shutdown_connections 2024-03-18T16:11:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:30.543+0000 7f84c60f7640 1 -- 172.21.15.150:0/1458249052 wait complete. 2024-03-18T16:11:31.343 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1166055 10.14 MB/sec execute 496 sec latency 51.318 ms 2024-03-18T16:11:31.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:31 smithi191 ceph-mon[36784]: pgmap v358: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 262 op/s 2024-03-18T16:11:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:31 smithi156 ceph-mon[35919]: pgmap v358: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 262 op/s 2024-03-18T16:11:32.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:31 smithi150 ceph-mon[31946]: pgmap v358: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 262 op/s 2024-03-18T16:11:32.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1168026 10.14 MB/sec execute 497 sec latency 29.042 ms 2024-03-18T16:11:33.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1169747 10.14 MB/sec execute 498 sec latency 26.610 ms 2024-03-18T16:11:33.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:33 smithi191 ceph-mon[36784]: pgmap v359: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:33 smithi156 ceph-mon[35919]: pgmap v359: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:33 smithi150 ceph-mon[31946]: pgmap v359: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:34.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1171576 10.14 MB/sec execute 499 sec latency 31.377 ms 2024-03-18T16:11:35.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1173098 10.13 MB/sec execute 500 sec latency 21.076 ms 2024-03-18T16:11:35.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:35 smithi191 ceph-mon[36784]: pgmap v360: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:35 smithi156 ceph-mon[35919]: pgmap v360: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:35 smithi150 ceph-mon[31946]: pgmap v360: 97 pgs: 97 active+clean; 655 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-03-18T16:11:36.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1174869 10.13 MB/sec execute 501 sec latency 98.088 ms 2024-03-18T16:11:37.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1176581 10.13 MB/sec execute 502 sec latency 24.424 ms 2024-03-18T16:11:37.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:37 smithi191 ceph-mon[36784]: pgmap v361: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 258 op/s 2024-03-18T16:11:38.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:37 smithi156 ceph-mon[35919]: pgmap v361: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 258 op/s 2024-03-18T16:11:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:37 smithi150 ceph-mon[31946]: pgmap v361: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 258 op/s 2024-03-18T16:11:38.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1178046 10.12 MB/sec execute 503 sec latency 25.396 ms 2024-03-18T16:11:39.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1179809 10.13 MB/sec execute 504 sec latency 22.461 ms 2024-03-18T16:11:39.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:39 smithi191 ceph-mon[36784]: pgmap v362: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 179 op/s 2024-03-18T16:11:40.024 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:39 smithi156 ceph-mon[35919]: pgmap v362: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 179 op/s 2024-03-18T16:11:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:39 smithi150 ceph-mon[31946]: pgmap v362: 97 pgs: 97 active+clean; 606 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 179 op/s 2024-03-18T16:11:40.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1181630 10.12 MB/sec execute 505 sec latency 31.206 ms 2024-03-18T16:11:41.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1183198 10.13 MB/sec execute 506 sec latency 76.398 ms 2024-03-18T16:11:41.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:41 smithi191 ceph-mon[36784]: pgmap v363: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 262 op/s 2024-03-18T16:11:42.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:41 smithi156 ceph-mon[35919]: pgmap v363: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 262 op/s 2024-03-18T16:11:42.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:41 smithi150 ceph-mon[31946]: pgmap v363: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 262 op/s 2024-03-18T16:11:42.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1184839 10.12 MB/sec execute 507 sec latency 19.139 ms 2024-03-18T16:11:43.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1186740 10.12 MB/sec execute 508 sec latency 16.558 ms 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.636+0000 7f97ccdb3640 1 -- 172.21.15.150:0/3906020340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81012f0 msgr2=0x7f97c810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.636+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/3906020340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81012f0 0x7f97c810d510 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009ca0 tx=0x7f97c002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 -- 172.21.15.150:0/3906020340 shutdown_connections 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/3906020340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c81012f0 0x7f97c810d510 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/3906020340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 0x7f97c8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.640 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/3906020340 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c8105680 0x7f97c8105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 -- 172.21.15.150:0/3906020340 >> 172.21.15.150:0/3906020340 conn(0x7f97c8077530 msgr2=0x7f97c8077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 -- 172.21.15.150:0/3906020340 shutdown_connections 2024-03-18T16:11:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.637+0000 7f97ccdb3640 1 -- 172.21.15.150:0/3906020340 wait complete. 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 Processor -- start 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 -- start start 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 0x7f97c8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c8105680 0x7f97c819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8110ba0 con 0x7f97c8105680 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8110a20 con 0x7f97c8100930 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97ccdb3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8110d20 con 0x7f97c81012f0 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97c5d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97c6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c8105680 0x7f97c819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97c5d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:39230/0 (socket says 172.21.15.150:39230) 2024-03-18T16:11:43.643 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.638+0000 7f97c5d9b640 1 -- 172.21.15.150:0/465608771 learned_addr learned my addr 172.21.15.150:0/465608771 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c659c640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 0x7f97c8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 -- 172.21.15.150:0/465608771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 msgr2=0x7f97c8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 0x7f97c8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 -- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c8105680 msgr2=0x7f97c819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c8105680 0x7f97c819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 -- 172.21.15.150:0/465608771 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c819cfe0 con 0x7f97c81012f0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009f20 tx=0x7f97bc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97b77fe640 1 -- 172.21.15.150:0/465608771 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc020070 con 0x7f97c81012f0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c819d2d0 con 0x7f97c81012f0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97b77fe640 1 -- 172.21.15.150:0/465608771 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97bc00cd80 con 0x7f97c81012f0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.639+0000 7f97b77fe640 1 -- 172.21.15.150:0/465608771 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc01bbb0 con 0x7f97c81012f0 2024-03-18T16:11:43.644 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.640+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c81074e0 con 0x7f97c81012f0 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.642+0000 7f97b77fe640 1 -- 172.21.15.150:0/465608771 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f97bc023070 con 0x7f97c81012f0 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.642+0000 7f97cf999740 1 Processor -- start 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97b77fe640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 0x7f97b00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 -- start start 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97b77fe640 1 -- 172.21.15.150:0/465608771 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f97bc0d2f70 con 0x7f97c81012f0 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eebfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 0x7f97b00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fb6ee18a40 con 0x55fb6eebd820 2024-03-18T16:11:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fb6edc1920 con 0x55fb6eeb4130 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97cf999740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fb6ee14ae0 con 0x55fb6eeb4a50 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33030/0 (socket says 172.21.15.150:33030) 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c659c640 1 -- 172.21.15.150:0/1936573057 learned_addr learned my addr 172.21.15.150:0/1936573057 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.643+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eebfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 -- 172.21.15.150:0/1936573057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 msgr2=0x55fb6eebd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 -- 172.21.15.150:0/1936573057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 msgr2=0x55fb6eebfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eebfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 -- 172.21.15.150:0/1936573057 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb6edd9460 con 0x55fb6eeb4130 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eebfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 0x7f97b00a4ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005cc0 tx=0x7f97c0032ea0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.648 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97c659c640 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009c80 tx=0x7f97b80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48781dcb69f91042 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97b4ff9640 1 -- 172.21.15.150:0/1936573057 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8012450 con 0x55fb6eeb4130 2024-03-18T16:11:43.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97b4ff9640 1 -- 172.21.15.150:0/1936573057 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b8012ad0 con 0x55fb6eeb4130 2024-03-18T16:11:43.649 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.644+0000 7f97b4ff9640 1 -- 172.21.15.150:0/1936573057 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b800fb20 con 0x55fb6eeb4130 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- 172.21.15.150:0/1936573057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 msgr2=0x55fb6eeb4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009c80 tx=0x7f97b80117b0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- 172.21.15.150:0/1936573057 shutdown_connections 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eebfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- 172.21.15.150:0/1936573057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeb4510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- 172.21.15.150:0/1936573057 >> 172.21.15.150:0/1936573057 conn(0x55fb6eeaea50 msgr2=0x55fb6eeaee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- 172.21.15.150:0/1936573057 shutdown_connections 2024-03-18T16:11:43.653 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- 172.21.15.150:0/1936573057 wait complete. 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- start start 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 Processor -- start 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eebd820 0x55fb6eebb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fb6eec0450 con 0x55fb6eeb4a50 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fb6edd4d70 con 0x55fb6eeb4130 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.645+0000 7f97cf999740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x55fb6eec05d0 con 0x55fb6eebd820 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97cf999740 10 client.0 _ll_register_callbacks cb 0x55fb6eb2ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.654 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33038/0 (socket says 172.21.15.150:33038) 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 -- 172.21.15.150:0/2957185719 learned_addr learned my addr 172.21.15.150:0/2957185719 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eebd820 0x55fb6eebb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 -- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eebd820 msgr2=0x55fb6eebb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eebd820 0x55fb6eebb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 -- 172.21.15.150:0/2957185719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 msgr2=0x55fb6eebad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 -- 172.21.15.150:0/2957185719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb6eb2d210 con 0x55fb6eeb4130 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97c659c640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f97b801c040 tx=0x7f97b8009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33fabf28568245d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97adffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8028020 con 0x55fb6eeb4130 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f97b8012450 con 0x55fb6eeb4130 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b8012ad0 con 0x55fb6eeb4130 2024-03-18T16:11:43.655 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97b8033050 con 0x55fb6eeb4130 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97cf999740 10 client.26138 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.646+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb6ebf9620 con 0x55fb6eeb4130 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b8012ad0 con 0x55fb6eeb4130 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f97b80035f0 con 0x55fb6eeb4130 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 10 client.26138 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 20 client.26138 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6eecd960 0x55fb6eecfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 4 client.26138 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb6ee25f80 con 0x55fb6eecd960 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6eecd960 0x55fb6eecfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6eecd960 0x55fb6eecfd50 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe48b280b49174c server_cookie=9611ca0bfb8af0f2 in_seq=0 out_seq=0 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.647+0000 7f97adffb640 10 client.26138 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97adffb640 1 -- 172.21.15.150:0/2957185719 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb6ee25f80 con 0x55fb6eecd960 2024-03-18T16:11:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97adffb640 10 client.26138 handle_command_reply: tid=0 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 client.26138 shutdown 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 msgr2=0x55fb6eeba840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f97b801c040 tx=0x7f97b8009fa0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 shutdown_connections 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6eecd960 0x55fb6eecfd50 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eebd820 0x55fb6eebb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eebad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97adffb640 0 client.26138 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 --2- 172.21.15.150:0/2957185719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eeba840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 >> 172.21.15.150:0/2957185719 conn(0x55fb6eeaea50 msgr2=0x55fb6eeb07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 shutdown_connections 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.648+0000 7f97cf999740 1 -- 172.21.15.150:0/2957185719 wait complete. 2024-03-18T16:11:43.657 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.649+0000 7f97cf999740 20 client.26138 trim_cache size 0 max 16384 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 Processor -- start 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 -- start start 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eebd820 0x55fb6eeb0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81ada50 con 0x55fb6eeb4a50 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81ab320 con 0x55fb6eebd820 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97cf999740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81a9190 con 0x55fb6eeb4130 2024-03-18T16:11:43.668 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97c659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.664+0000 7f97c5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:39258/0 (socket says 172.21.15.150:39258) 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 -- 172.21.15.150:0/3364440277 learned_addr learned my addr 172.21.15.150:0/3364440277 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eebd820 0x55fb6eeb0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 -- 172.21.15.150:0/3364440277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eebd820 msgr2=0x55fb6eeb0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eebd820 0x55fb6eeb0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 -- 172.21.15.150:0/3364440277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 msgr2=0x55fb6eec91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 -- 172.21.15.150:0/3364440277 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb6eeae880 con 0x55fb6eeb4130 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97c659c640 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f97b800f6d0 tx=0x7f97b8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3429f9c28ef17287 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97affff640 1 -- 172.21.15.150:0/3364440277 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b803b020 con 0x55fb6eeb4130 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97affff640 1 -- 172.21.15.150:0/3364440277 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b8009590 con 0x55fb6eeb4130 2024-03-18T16:11:43.669 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.665+0000 7f97affff640 1 -- 172.21.15.150:0/3364440277 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b8028040 con 0x55fb6eeb4130 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- 172.21.15.150:0/3364440277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 msgr2=0x55fb6eebcf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f97b800f6d0 tx=0x7f97b8009260 comp rx=0 tx=0).stop 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- 172.21.15.150:0/3364440277 shutdown_connections 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eebd820 0x55fb6eeb0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 --2- 172.21.15.150:0/3364440277 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4130 0x55fb6eebcf70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- 172.21.15.150:0/3364440277 >> 172.21.15.150:0/3364440277 conn(0x55fb6eeaea50 msgr2=0x55fb6eeafe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- 172.21.15.150:0/3364440277 shutdown_connections 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- 172.21.15.150:0/3364440277 wait complete. 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.666+0000 7f97cf999740 1 -- start start 2024-03-18T16:11:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 Processor -- start 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eec1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81a97e0 con 0x55fb6eebd820 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.150:33058/0 (socket says 172.21.15.150:33058) 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c659c640 1 -- 172.21.15.150:0/947615768 learned_addr learned my addr 172.21.15.150:0/947615768 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81aac50 con 0x55fb6eeb4130 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f97c81a9eb0 con 0x55fb6eeb4a50 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eec1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97cf999740 10 client.0 _ll_register_callbacks cb 0x7f97bc01b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 -- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 msgr2=0x55fb6eec0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 -- 172.21.15.150:0/947615768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 msgr2=0x55fb6eec5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c6d9d640 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb6eeb3720 con 0x55fb6eebd820 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c5d9b640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.667+0000 7f97c659c640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97c6d9d640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eec1320 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f97c002c630 tx=0x7f97c0045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b529433c7b582c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97adffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0039040 con 0x55fb6eebd820 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f97c0004450 con 0x55fb6eebd820 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb6ee202d0 con 0x55fb6eebd820 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97c005d050 con 0x55fb6eebd820 2024-03-18T16:11:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97cf999740 10 client.16380 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.668+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb6eebfc10 con 0x55fb6eebd820 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb6ee202d0 con 0x55fb6eebd820 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f97c004d070 con 0x55fb6eebd820 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 10 client.16380 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 10 client.16380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 20 client.16380 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6ef33d20 0x55fb6ef36140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 4 client.16380 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb6ed9e770 con 0x55fb6ef33d20 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97c659c640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6ef33d20 0x55fb6ef36140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97c659c640 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6ef33d20 0x55fb6ef36140 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=477c957cea837d0f server_cookie=1b4fe0c4451be3f8 in_seq=0 out_seq=0 2024-03-18T16:11:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.669+0000 7f97adffb640 10 client.16380 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97adffb640 1 -- 172.21.15.150:0/947615768 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb6ed9e770 con 0x55fb6ef33d20 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97adffb640 10 client.16380 handle_command_reply: tid=0 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 client.16380 shutdown 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 msgr2=0x55fb6eec1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eec1320 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f97c002c630 tx=0x7f97c0045070 comp rx=0 tx=0).stop 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 shutdown_connections 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x55fb6ef33d20 0x55fb6ef36140 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb6eebd820 0x55fb6eec1320 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97adffb640 0 client.16380 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:43.685 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55fb6eeb4a50 0x55fb6eec0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 --2- 172.21.15.150:0/947615768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb6eeb4130 0x55fb6eec5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 >> 172.21.15.150:0/947615768 conn(0x55fb6eeaea50 msgr2=0x55fb6eeadbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 shutdown_connections 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 1 -- 172.21.15.150:0/947615768 wait complete. 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.682+0000 7f97cf999740 20 client.16380 trim_cache size 0 max 16384 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:11:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 msgr2=0x7f97b00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 0x7f97b00a4ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005cc0 tx=0x7f97c0032ea0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 msgr2=0x7f97c8198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009f20 tx=0x7f97bc00ee80 comp rx=0 tx=0).stop 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 shutdown_connections 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f97b00a2a10 0x7f97b00a4ed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97c8105680 0x7f97c819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c81012f0 0x7f97c8198530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 --2- 172.21.15.150:0/465608771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100930 0x7f97c8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 >> 172.21.15.150:0/465608771 conn(0x7f97c8077530 msgr2=0x7f97c80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 shutdown_connections 2024-03-18T16:11:43.688 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:43.683+0000 7f97ccdb3640 1 -- 172.21.15.150:0/465608771 wait complete. 2024-03-18T16:11:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:43 smithi156 ceph-mon[35919]: pgmap v364: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:43 smithi150 ceph-mon[31946]: pgmap v364: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:44.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:43 smithi191 ceph-mon[36784]: pgmap v364: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:44.344 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1188491 10.11 MB/sec execute 509 sec latency 24.837 ms 2024-03-18T16:11:45.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1190432 10.12 MB/sec execute 510 sec latency 31.613 ms 2024-03-18T16:11:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:45 smithi156 ceph-mon[35919]: pgmap v365: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:46.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:45 smithi150 ceph-mon[31946]: pgmap v365: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:45 smithi191 ceph-mon[36784]: pgmap v365: 97 pgs: 97 active+clean; 617 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-03-18T16:11:46.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1191931 10.11 MB/sec execute 511 sec latency 112.584 ms 2024-03-18T16:11:47.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1193803 10.12 MB/sec execute 512 sec latency 37.616 ms 2024-03-18T16:11:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:47 smithi156 ceph-mon[35919]: pgmap v366: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 259 op/s 2024-03-18T16:11:48.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:47 smithi150 ceph-mon[31946]: pgmap v366: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 259 op/s 2024-03-18T16:11:48.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:47 smithi191 ceph-mon[36784]: pgmap v366: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 259 op/s 2024-03-18T16:11:48.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1194972 10.11 MB/sec execute 513 sec latency 21.637 ms 2024-03-18T16:11:49.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1197178 10.12 MB/sec execute 514 sec latency 18.447 ms 2024-03-18T16:11:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:49 smithi156 ceph-mon[35919]: pgmap v367: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:11:50.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:49 smithi150 ceph-mon[31946]: pgmap v367: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:11:50.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:49 smithi191 ceph-mon[36784]: pgmap v367: 97 pgs: 97 active+clean; 629 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-03-18T16:11:50.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1198625 10.11 MB/sec execute 515 sec latency 20.786 ms 2024-03-18T16:11:51.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1200683 10.11 MB/sec execute 516 sec latency 107.506 ms 2024-03-18T16:11:52.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:51 smithi156 ceph-mon[35919]: pgmap v368: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 257 op/s 2024-03-18T16:11:52.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:51 smithi150 ceph-mon[31946]: pgmap v368: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 257 op/s 2024-03-18T16:11:52.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:51 smithi191 ceph-mon[36784]: pgmap v368: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 257 op/s 2024-03-18T16:11:52.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1202039 10.11 MB/sec execute 517 sec latency 25.787 ms 2024-03-18T16:11:53.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1204262 10.11 MB/sec execute 518 sec latency 18.436 ms 2024-03-18T16:11:54.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:53 smithi156 ceph-mon[35919]: pgmap v369: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:54.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:53 smithi150 ceph-mon[31946]: pgmap v369: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:54.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:53 smithi191 ceph-mon[36784]: pgmap v369: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:54.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1205679 10.10 MB/sec execute 519 sec latency 19.849 ms 2024-03-18T16:11:55.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1207819 10.11 MB/sec execute 520 sec latency 36.025 ms 2024-03-18T16:11:56.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:55 smithi156 ceph-mon[35919]: pgmap v370: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:56.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:55 smithi150 ceph-mon[31946]: pgmap v370: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:56.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:55 smithi191 ceph-mon[36784]: pgmap v370: 97 pgs: 97 active+clean; 639 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 173 op/s 2024-03-18T16:11:56.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1209069 10.10 MB/sec execute 521 sec latency 89.067 ms 2024-03-18T16:11:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 -- 172.21.15.150:0/229888385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc101790 msgr2=0x7fb0bc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/229888385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc101790 0x7fb0bc101c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009930 tx=0x7fb0ac02bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 -- 172.21.15.150:0/229888385 shutdown_connections 2024-03-18T16:11:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/229888385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc102150 0x7fb0bc10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/229888385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc101790 0x7fb0bc101c10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/229888385 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 -- 172.21.15.150:0/229888385 >> 172.21.15.150:0/229888385 conn(0x7fb0bc0fd2a0 msgr2=0x7fb0bc0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 -- 172.21.15.150:0/229888385 shutdown_connections 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.775+0000 7fb0c37e6640 1 -- 172.21.15.150:0/229888385 wait complete. 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 Processor -- start 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 -- start start 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc101790 0x7fb0bc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 0x7fb0bc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc111bb0 con 0x7fb0bc1077a0 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc111a30 con 0x7fb0bc102150 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c37e6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc111d30 con 0x7fb0bc101790 2024-03-18T16:11:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53568/0 (socket says 172.21.15.150:53568) 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 -- 172.21.15.150:0/3137936440 learned_addr learned my addr 172.21.15.150:0/3137936440 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1582640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc101790 0x7fb0bc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 0x7fb0bc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 -- 172.21.15.150:0/3137936440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc101790 msgr2=0x7fb0bc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc101790 0x7fb0bc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 -- 172.21.15.150:0/3137936440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 msgr2=0x7fb0bc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 0x7fb0bc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c1d83640 1 -- 172.21.15.150:0/3137936440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc19d020 con 0x7fb0bc1077a0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.776+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 0x7fb0bc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009fd0 tx=0x7fb0b400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0b27fc640 1 -- 172.21.15.150:0/3137936440 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4017070 con 0x7fb0bc1077a0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0b27fc640 1 -- 172.21.15.150:0/3137936440 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b4003e50 con 0x7fb0bc1077a0 2024-03-18T16:11:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc19d310 con 0x7fb0bc1077a0 2024-03-18T16:11:56.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0b27fc640 1 -- 172.21.15.150:0/3137936440 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40106c0 con 0x7fb0bc1077a0 2024-03-18T16:11:56.782 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.777+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc1a4ba0 con 0x7fb0bc1077a0 2024-03-18T16:11:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.779+0000 7fb0b27fc640 1 -- 172.21.15.150:0/3137936440 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fb0b4003970 con 0x7fb0bc1077a0 2024-03-18T16:11:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.779+0000 7fb0c63d2740 1 Processor -- start 2024-03-18T16:11:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 -- start start 2024-03-18T16:11:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0b27fc640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 0x7fb0a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555e70ebda40 con 0x555e70f59130 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555e70e66920 con 0x555e70f59a50 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555e70eb9ae0 con 0x555e70f62820 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 0x7fb0a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53574/0 (socket says 172.21.15.150:53574) 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 -- 172.21.15.150:0/160852750 learned_addr learned my addr 172.21.15.150:0/160852750 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0b27fc640 1 -- 172.21.15.150:0/3137936440 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7fb0b40ceb70 con 0x7fb0bc1077a0 2024-03-18T16:11:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 -- 172.21.15.150:0/160852750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 msgr2=0x555e70f64c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f64c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 -- 172.21.15.150:0/160852750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 msgr2=0x555e70f622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 -- 172.21.15.150:0/160852750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e70e7e460 con 0x555e70f59130 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c1582640 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009c70 tx=0x7fb0b8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694b67af05dc39d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.780+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 0x7fb0a80a4ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005ab0 tx=0x7fb0ac005e80 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0a77fe640 1 -- 172.21.15.150:0/160852750 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8004230 con 0x555e70f59130 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0a77fe640 1 -- 172.21.15.150:0/160852750 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b80043d0 con 0x555e70f59130 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0a77fe640 1 -- 172.21.15.150:0/160852750 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8019760 con 0x555e70f59130 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 -- 172.21.15.150:0/160852750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 msgr2=0x555e70f59510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009c70 tx=0x7fb0b8011810 comp rx=0 tx=0).stop 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 -- 172.21.15.150:0/160852750 shutdown_connections 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/160852750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59130 0x555e70f59510 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 -- 172.21.15.150:0/160852750 >> 172.21.15.150:0/160852750 conn(0x555e70f53a50 msgr2=0x555e70f53e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 -- 172.21.15.150:0/160852750 shutdown_connections 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.781+0000 7fb0c63d2740 1 -- 172.21.15.150:0/160852750 wait complete. 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 -- start start 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 Processor -- start 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f5f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f60260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x555e70f65450 con 0x555e70f59a50 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555e70e79d70 con 0x555e70f59130 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x555e70f655d0 con 0x555e70f62820 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c63d2740 10 client.0 _ll_register_callbacks cb 0x555e70bd1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c0d81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c1582640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f5f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c0d81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53590/0 (socket says 172.21.15.150:53590) 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c0d81640 1 -- 172.21.15.150:0/3126897678 learned_addr learned my addr 172.21.15.150:0/3126897678 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.782+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f60260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 -- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 msgr2=0x555e70f60260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f60260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 -- 172.21.15.150:0/3126897678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 msgr2=0x555e70f5f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.792 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f5f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 -- 172.21.15.150:0/3126897678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e70bd2210 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005db0 tx=0x7fb0ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88e467387313891 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac005140 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7fb0ac0052e0 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b80043d0 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb0ac0054e0 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c63d2740 10 client.16392 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555e70c9e620 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b80043d0 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.783+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fb0ac0436b0 con 0x555e70f59a50 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 10 client.16392 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 10 client.16392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 20 client.16392 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:56.793 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70f72980 0x555e70f74dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 4 client.16392 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e70ecaf80 con 0x555e70f72980 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c1582640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70f72980 0x555e70f74dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0c1582640 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70f72980 0x555e70f74dd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686d4bd99d69ecbb server_cookie=332ffca45a27ecdb in_seq=0 out_seq=0 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.784+0000 7fb0a4ff9640 10 client.16392 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/3126897678 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555e70ecaf80 con 0x555e70f72980 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0a4ff9640 10 client.16392 handle_command_reply: tid=0 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 client.16392 shutdown 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 msgr2=0x555e70f5fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.794 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005db0 tx=0x7fb0ac02bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 shutdown_connections 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70f72980 0x555e70f74dd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f62820 0x555e70f60260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f59a50 0x555e70f5fd20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/3126897678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f5f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0a4ff9640 0 client.16392 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 >> 172.21.15.150:0/3126897678 conn(0x555e70f53a50 msgr2=0x555e70f557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 shutdown_connections 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 1 -- 172.21.15.150:0/3126897678 wait complete. 2024-03-18T16:11:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.785+0000 7fb0c63d2740 20 client.16392 trim_cache size 0 max 16384 2024-03-18T16:11:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 Processor -- start 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 -- start start 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f6e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59a50 0x555e70f7a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1afbc0 con 0x555e70f62820 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1ad490 con 0x555e70f59130 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1ab300 con 0x555e70f59a50 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.800+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53602/0 (socket says 172.21.15.150:53602) 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2382393851 learned_addr learned my addr 172.21.15.150:0/2382393851 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1582640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f6e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59a50 0x555e70f7a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2382393851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59a50 msgr2=0x555e70f7a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59a50 0x555e70f7a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2382393851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 msgr2=0x555e70f6e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f6e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2382393851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e70f59880 con 0x555e70f62820 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1582640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f6e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb0b409af80 tx=0x7fb0b40044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12286ecff099e1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0a6ffd640 1 -- 172.21.15.150:0/2382393851 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4020040 con 0x555e70f62820 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0a6ffd640 1 -- 172.21.15.150:0/2382393851 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b404ee50 con 0x555e70f62820 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.801+0000 7fb0a6ffd640 1 -- 172.21.15.150:0/2382393851 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40040b0 con 0x555e70f62820 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2382393851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 msgr2=0x555e70f6a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb0b409af80 tx=0x7fb0b40044b0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2382393851 shutdown_connections 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f6a810 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59a50 0x555e70f7a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2382393851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59130 0x555e70f6e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2382393851 >> 172.21.15.150:0/2382393851 conn(0x555e70f53a50 msgr2=0x555e70f52320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2382393851 shutdown_connections 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2382393851 wait complete. 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 -- start start 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 Processor -- start 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59130 0x555e70f6a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.802+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1ab950 con 0x555e70f62820 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1acdc0 con 0x555e70f59a50 2024-03-18T16:11:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1ac020 con 0x555e70f59130 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 10 client.0 _ll_register_callbacks cb 0x7fb0ac005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1582640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59130 0x555e70f6a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c0d81640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53614/0 (socket says 172.21.15.150:53614) 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2343084474 learned_addr learned my addr 172.21.15.150:0/2343084474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59130 msgr2=0x555e70f6a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59130 0x555e70f6a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2343084474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 msgr2=0x555e70f677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 -- 172.21.15.150:0/2343084474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e70f6af10 con 0x555e70f62820 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c0d81640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c1d83640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400b760 tx=0x7fb0b4099790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d9664f9032998b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4020070 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7fb0b40cc540 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b404ee50 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb0b409f320 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 10 client.16398 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.803+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555e70fd97d0 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b404ee50 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fb0b409a7f0 con 0x555e70f62820 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 10 client.16398 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 10 client.16398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 20 client.16398 populate_metadata read hostname 'smithi150' 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70fd9e90 0x555e70fdc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 4 client.16398 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 --> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e70ec52d0 con 0x555e70fd9e90 2024-03-18T16:11:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c1582640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70fd9e90 0x555e70fdc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0a4ff9640 10 client.16398 ms_handle_connect on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.804+0000 7fb0c1582640 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70fd9e90 0x555e70fdc2b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aee3f0ec322f28cd server_cookie=b2684b7af0c886bf in_seq=0 out_seq=0 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0a4ff9640 1 -- 172.21.15.150:0/2343084474 <== mds.0 v2:172.21.15.191:6838/3948564183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e70ec52d0 con 0x555e70fd9e90 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0a4ff9640 10 client.16398 handle_command_reply: tid=0 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 client.16398 shutdown 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 msgr2=0x555e70f67d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400b760 tx=0x7fb0b4099790 comp rx=0 tx=0).stop 2024-03-18T16:11:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 shutdown_connections 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:6838/3948564183,v1:172.21.15.191:6839/3948564183] conn(0x555e70fd9e90 0x555e70fdc2b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555e70f62820 0x555e70f67d30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555e70f59a50 0x555e70f677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0a4ff9640 0 client.16398 ms_handle_reset on v2:172.21.15.191:6838/3948564183 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 --2- 172.21.15.150:0/2343084474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555e70f59130 0x555e70f6a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 >> 172.21.15.150:0/2343084474 conn(0x555e70f53a50 msgr2=0x555e70f52bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 shutdown_connections 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 1 -- 172.21.15.150:0/2343084474 wait complete. 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.817+0000 7fb0c63d2740 20 client.16398 trim_cache size 0 max 16384 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:{ 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "message": "", 2024-03-18T16:11:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout: "return_code": 0 2024-03-18T16:11:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stdout:} 2024-03-18T16:11:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 msgr2=0x7fb0a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 0x7fb0a80a4ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005ab0 tx=0x7fb0ac005e80 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 msgr2=0x7fb0bc19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009fd0 tx=0x7fb0b400b260 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 shutdown_connections 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fb0a80a2a10 0x7fb0a80a4ed0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc1077a0 0x7fb0bc19c8f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0bc102150 0x7fb0bc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 --2- 172.21.15.150:0/3137936440 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0bc101790 0x7fb0bc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 >> 172.21.15.150:0/3137936440 conn(0x7fb0bc0fd2a0 msgr2=0x7fb0bc0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 shutdown_connections 2024-03-18T16:11:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi150.front.sepia.ceph.com.smithi150.stderr:2024-03-18T16:11:56.818+0000 7fb0c37e6640 1 -- 172.21.15.150:0/3137936440 wait complete. 2024-03-18T16:11:57.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1210912 10.10 MB/sec execute 522 sec latency 145.393 ms 2024-03-18T16:11:58.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:57 smithi156 ceph-mon[35919]: pgmap v371: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 263 op/s 2024-03-18T16:11:58.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:57 smithi150 ceph-mon[31946]: pgmap v371: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 263 op/s 2024-03-18T16:11:58.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:57 smithi191 ceph-mon[36784]: pgmap v371: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 263 op/s 2024-03-18T16:11:58.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1212364 10.09 MB/sec execute 523 sec latency 21.206 ms 2024-03-18T16:11:59.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1214468 10.10 MB/sec execute 524 sec latency 19.271 ms 2024-03-18T16:12:00.023 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:11:59 smithi156 ceph-mon[35919]: pgmap v372: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 174 op/s 2024-03-18T16:12:00.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:11:59 smithi150 ceph-mon[31946]: pgmap v372: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 174 op/s 2024-03-18T16:12:00.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:11:59 smithi191 ceph-mon[36784]: pgmap v372: 97 pgs: 97 active+clean; 650 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.4 MiB/s wr, 174 op/s 2024-03-18T16:12:00.345 INFO:tasks.workunit.client.0.smithi150.stdout: 1 1215900 10.09 MB/sec execute 525 sec latency 21.613 ms 2024-03-18T16:12:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:12:00.398 INFO:tasks.workunit.client.0.smithi150.stdout:[1215933] write failed on handle 12369 (Resource temporarily unavailable) 2024-03-18T16:12:00.398 INFO:tasks.workunit.client.0.smithi150.stdout:Child failed with status 1 2024-03-18T16:12:00.398 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-18T16:12:00.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T16:12:00.683 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-18T16:12:00.964 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=32e48465455c44c29ff08564d4078e54 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-18T16:12:00.967 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-18T16:12:00.980 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T16:12:00.993 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-18T16:12:01.004 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-18T16:12:01.004 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T16:12:01.004 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T16:12:01.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-18T16:12:01.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-18T16:12:01.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-18T16:12:01.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- 172.21.15.150:0/1997483062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a9c0 msgr2=0x7fef3c10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 --2- 172.21.15.150:0/1997483062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a9c0 0x7fef3c10e600 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fef34009ca0 tx=0x7fef3402c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- 172.21.15.150:0/1997483062 shutdown_connections 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 --2- 172.21.15.150:0/1997483062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a9c0 0x7fef3c10e600 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 --2- 172.21.15.150:0/1997483062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a020 0x7fef3c06a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 --2- 172.21.15.150:0/1997483062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 0x7fef3c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- 172.21.15.150:0/1997483062 >> 172.21.15.150:0/1997483062 conn(0x7fef3c077400 msgr2=0x7fef3c077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- 172.21.15.150:0/1997483062 shutdown_connections 2024-03-18T16:12:01.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- 172.21.15.150:0/1997483062 wait complete. 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 Processor -- start 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.280+0000 7fef42e20640 1 -- start start 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a9c0 0x7fef3c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 0x7fef3c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c111ba0 con 0x7fef3c06a020 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c111a20 con 0x7fef3c1077b0 2024-03-18T16:12:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c111d20 con 0x7fef3c06a9c0 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53628/0 (socket says 172.21.15.150:53628) 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 -- 172.21.15.150:0/708611602 learned_addr learned my addr 172.21.15.150:0/708611602 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef3bfff640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a9c0 0x7fef3c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef413bd640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 0x7fef3c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 -- 172.21.15.150:0/708611602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a9c0 msgr2=0x7fef3c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a9c0 0x7fef3c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 -- 172.21.15.150:0/708611602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 msgr2=0x7fef3c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 0x7fef3c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 -- 172.21.15.150:0/708611602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c19cf30 con 0x7fef3c06a020 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef40bbc640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fef3000e9f0 tx=0x7fef3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3000cd80 con 0x7fef3c06a020 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3c19d220 con 0x7fef3c06a020 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef30010910 con 0x7fef3c06a020 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.281+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300188f0 con 0x7fef3c06a020 2024-03-18T16:12:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.282+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3c1a4ab0 con 0x7fef3c06a020 2024-03-18T16:12:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.283+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3c102ed0 con 0x7fef3c06a020 2024-03-18T16:12:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.284+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fef30010ac0 con 0x7fef3c06a020 2024-03-18T16:12:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.284+0000 7fef39ffb640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 0x7fef2c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.284+0000 7fef3bfff640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 0x7fef2c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.284+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7fef300cf0d0 con 0x7fef3c06a020 2024-03-18T16:12:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.285+0000 7fef3bfff640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 0x7fef2c0a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef3c106d10 tx=0x7fef28005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.289+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef3009b880 con 0x7fef3c06a020 2024-03-18T16:12:01.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.506+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fef3c0ffb80 con 0x7fef2c0a2a10 2024-03-18T16:12:01.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.515+0000 7fef39ffb640 1 -- 172.21.15.150:0/708611602 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fef3c0ffb80 con 0x7fef2c0a2a10 2024-03-18T16:12:01.517 INFO:teuthology.orchestra.run.smithi150.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T16:00:05", "first": "2024-03-18T16:01:00", "last": "2024-03-18T16:12:00", "last_pruned": "2024-03-18T16:12:00", "created_count": 12, "pruned_count": 6, "active": true} 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 msgr2=0x7fef2c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 0x7fef2c0a4ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef3c106d10 tx=0x7fef28005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 msgr2=0x7fef3c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fef3000e9f0 tx=0x7fef3000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 shutdown_connections 2024-03-18T16:12:01.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fef2c0a2a10 0x7fef2c0a4ed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef3c1077b0 0x7fef3c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef3c06a9c0 0x7fef3c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 --2- 172.21.15.150:0/708611602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c06a020 0x7fef3c197f40 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 >> 172.21.15.150:0/708611602 conn(0x7fef3c077400 msgr2=0x7fef3c0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 shutdown_connections 2024-03-18T16:12:01.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.517+0000 7fef42e20640 1 -- 172.21.15.150:0/708611602 wait complete. 2024-03-18T16:12:01.530 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-18T16:12:01.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- 172.21.15.150:0/4158619273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 msgr2=0x7f018c105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 --2- 172.21.15.150:0/4158619273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c105a60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0184009900 tx=0x7f018402bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:01.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- 172.21.15.150:0/4158619273 shutdown_connections 2024-03-18T16:12:01.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 --2- 172.21.15.150:0/4158619273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 0x7f018c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 --2- 172.21.15.150:0/4158619273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 --2- 172.21.15.150:0/4158619273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c105a60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- 172.21.15.150:0/4158619273 >> 172.21.15.150:0/4158619273 conn(0x7f018c077530 msgr2=0x7f018c077940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- 172.21.15.150:0/4158619273 shutdown_connections 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- 172.21.15.150:0/4158619273 wait complete. 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 Processor -- start 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.632+0000 7f01935b2640 1 -- start start 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 0x7f018c19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f018c069720 con 0x7f018c105680 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0695a0 con 0x7f018c100930 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0698a0 con 0x7f018c1012f0 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f019134e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53632/0 (socket says 172.21.15.150:53632) 2024-03-18T16:12:01.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 -- 172.21.15.150:0/1372711403 learned_addr learned my addr 172.21.15.150:0/1372711403 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0190b4d640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 0x7f018c19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 -- 172.21.15.150:0/1372711403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 msgr2=0x7f018c19f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 0x7f018c19f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 -- 172.21.15.150:0/1372711403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 msgr2=0x7f018c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 -- 172.21.15.150:0/1372711403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c10ce10 con 0x7f018c105680 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f019134e640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f0191b4f640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f017c00ea60 tx=0x7f017c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c00ce40 con 0x7f018c105680 2024-03-18T16:12:01.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f017c005020 con 0x7f018c105680 2024-03-18T16:12:01.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c10d100 con 0x7f018c105680 2024-03-18T16:12:01.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c010690 con 0x7f018c105680 2024-03-18T16:12:01.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.633+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c1a6fe0 con 0x7f018c105680 2024-03-18T16:12:01.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.634+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c101df0 con 0x7f018c105680 2024-03-18T16:12:01.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.635+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f017c005520 con 0x7f018c105680 2024-03-18T16:12:01.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.636+0000 7f018a7fc640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 0x7f01740a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.636+0000 7f019134e640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 0x7f01740a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.636+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f017c0cf520 con 0x7f018c105680 2024-03-18T16:12:01.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.636+0000 7f019134e640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 0x7f01740a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0184009230 tx=0x7f0184005fd0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.640+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f017c053740 con 0x7f018c105680 2024-03-18T16:12:01.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.840+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 --> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f018c111880 con 0x7f01740a2a10 2024-03-18T16:12:01.842 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:01 smithi150 ceph-mon[31946]: pgmap v373: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:12:01.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.848+0000 7f018a7fc640 1 -- 172.21.15.150:0/1372711403 <== mgr.15336 v2:172.21.15.150:6800/669533394 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f018c111880 con 0x7f01740a2a10 2024-03-18T16:12:01.849 INFO:teuthology.orchestra.run.smithi150.stdout:/ 1m 6m3h 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 msgr2=0x7f01740a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 0x7f01740a4ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0184009230 tx=0x7f0184005fd0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 msgr2=0x7f018c10c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f017c00ea60 tx=0x7f017c00ef20 comp rx=0 tx=0).stop 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 shutdown_connections 2024-03-18T16:12:01.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f01740a2a10 0x7f01740a4ed0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f018c105680 0x7f018c10c550 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f018c1012f0 0x7f018c19f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 --2- 172.21.15.150:0/1372711403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f018c100930 0x7f018c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 >> 172.21.15.150:0/1372711403 conn(0x7f018c077530 msgr2=0x7f018c111170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 shutdown_connections 2024-03-18T16:12:01.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.850+0000 7f01935b2640 1 -- 172.21.15.150:0/1372711403 wait complete. 2024-03-18T16:12:01.863 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T16:12:01.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:12:01.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 -- 172.21.15.150:0/247980050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 msgr2=0x7f834810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 --2- 172.21.15.150:0/247980050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834810d570 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8340009cd0 tx=0x7f834002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 -- 172.21.15.150:0/247980050 shutdown_connections 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 --2- 172.21.15.150:0/247980050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834810d570 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 --2- 172.21.15.150:0/247980050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 0x7f8348100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 --2- 172.21.15.150:0/247980050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 0x7f8348106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 -- 172.21.15.150:0/247980050 >> 172.21.15.150:0/247980050 conn(0x7f8348077780 msgr2=0x7f8348077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 -- 172.21.15.150:0/247980050 shutdown_connections 2024-03-18T16:12:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 -- 172.21.15.150:0/247980050 wait complete. 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.974+0000 7f834d085640 1 Processor -- start 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 -- start start 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 0x7f834819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 0x7f834819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8348111bc0 con 0x7f8348101060 2024-03-18T16:12:01.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8348111a40 con 0x7f83481006a0 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f8348111d40 con 0x7f83481066b0 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53652/0 (socket says 172.21.15.150:53652) 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 -- 172.21.15.150:0/2146873640 learned_addr learned my addr 172.21.15.150:0/2146873640 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834759e640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 0x7f834819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f8346d9d640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 0x7f834819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 -- 172.21.15.150:0/2146873640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 msgr2=0x7f834819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 0x7f834819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 -- 172.21.15.150:0/2146873640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 msgr2=0x7f834819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 0x7f834819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 -- 172.21.15.150:0/2146873640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834819f290 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834659c640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f833800e9f0 tx=0x7f833800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833800cd80 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338010910 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338018990 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.975+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834819f580 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.976+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83481a6e10 con 0x7f8348101060 2024-03-18T16:12:01.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.976+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8348068a20 con 0x7f8348101060 2024-03-18T16:12:01.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.978+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f8338010ac0 con 0x7f8348101060 2024-03-18T16:12:01.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.978+0000 7f832ffff640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 0x7f83300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:01.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.978+0000 7f8346d9d640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 0x7f83300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:01.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.978+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f83380cf580 con 0x7f8348101060 2024-03-18T16:12:01.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.979+0000 7f8346d9d640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 0x7f83300a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f833c009b50 tx=0x7f833c005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:01.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:01.982+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f833809bc90 con 0x7f8348101060 2024-03-18T16:12:02.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:01 smithi156 ceph-mon[35919]: pgmap v373: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:12:02.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.159+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83481a7040 con 0x7f8348101060 2024-03-18T16:12:02.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.160+0000 7f832ffff640 1 -- 172.21.15.150:0/2146873640 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+23202 (secure 0 0 0) 0x7f833809fd30 con 0x7f8348101060 2024-03-18T16:12:02.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:02.163 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T16:11:09.413776+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2222781423":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6800/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/1896654966":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/3130851108":"2024-03-18T17:04:57.464453+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3143158665":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3506843134":"2024-03-19T16:00:04.468782+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:12:02.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 msgr2=0x7f83300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 0x7f83300a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f833c009b50 tx=0x7f833c005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:02.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 msgr2=0x7f834819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f833800e9f0 tx=0x7f833800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 shutdown_connections 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f83300a2970 0x7f83300a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83481066b0 0x7f834819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8348101060 0x7f834819a7e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 --2- 172.21.15.150:0/2146873640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83481006a0 0x7f834819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 >> 172.21.15.150:0/2146873640 conn(0x7f8348077780 msgr2=0x7f834810b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 shutdown_connections 2024-03-18T16:12:02.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.162+0000 7f834d085640 1 -- 172.21.15.150:0/2146873640 wait complete. 2024-03-18T16:12:02.176 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:12:02.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:12:02.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:01 smithi191 ceph-mon[36784]: pgmap v373: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.2 MiB/s wr, 281 op/s 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- 172.21.15.150:0/2211544938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 msgr2=0x7f3b10073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2211544938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b10073f90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009ca0 tx=0x7f3b0c02c180 comp rx=0 tx=0).stop 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- 172.21.15.150:0/2211544938 shutdown_connections 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2211544938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 0x7f3b10110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2211544938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 0x7f3b1010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2211544938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b10073f90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- 172.21.15.150:0/2211544938 >> 172.21.15.150:0/2211544938 conn(0x7f3b10078be0 msgr2=0x7f3b10078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- 172.21.15.150:0/2211544938 shutdown_connections 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- 172.21.15.150:0/2211544938 wait complete. 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 Processor -- start 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.281+0000 7f3b17c54640 1 -- start start 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 0x7f3b1019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 0x7f3b1019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b10113d90 con 0x7f3b10073bb0 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b10113c10 con 0x7f3b100744d0 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3b10113f10 con 0x7f3b1010e010 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53670/0 (socket says 172.21.15.150:53670) 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 -- 172.21.15.150:0/2870325848 learned_addr learned my addr 172.21.15.150:0/2870325848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b151ef640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 0x7f3b1019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b161f1640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 0x7f3b1019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 -- 172.21.15.150:0/2870325848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 msgr2=0x7f3b1019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 0x7f3b1019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 -- 172.21.15.150:0/2870325848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 msgr2=0x7f3b1019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 0x7f3b1019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 -- 172.21.15.150:0/2870325848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1019f2e0 con 0x7f3b10073bb0 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b159f0640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c005e30 tx=0x7f3b0c035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c039040 con 0x7f3b10073bb0 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1019f570 con 0x7f3b10073bb0 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b0c044070 con 0x7f3b10073bb0 2024-03-18T16:12:02.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c0328d0 con 0x7f3b10073bb0 2024-03-18T16:12:02.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.282+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b101a6e60 con 0x7f3b10073bb0 2024-03-18T16:12:02.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.283+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b10068a20 con 0x7f3b10073bb0 2024-03-18T16:12:02.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.285+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f3b0c032a70 con 0x7f3b10073bb0 2024-03-18T16:12:02.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.285+0000 7f3b06ffd640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 0x7f3afc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.285+0000 7f3b151ef640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 0x7f3afc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.285+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f3b0c0f11e0 con 0x7f3b10073bb0 2024-03-18T16:12:02.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.286+0000 7f3b151ef640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 0x7f3afc0a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3b10100970 tx=0x7f3b00005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.290+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b0c0be050 con 0x7f3b10073bb0 2024-03-18T16:12:02.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.482+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b10109190 con 0x7f3b10073bb0 2024-03-18T16:12:02.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.482+0000 7f3b06ffd640 1 -- 172.21.15.150:0/2870325848 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f3b0c0bd950 con 0x7f3b10073bb0 2024-03-18T16:12:02.484 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 msgr2=0x7f3afc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 0x7f3afc0a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3b10100970 tx=0x7f3b00005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 msgr2=0x7f3b1019a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c005e30 tx=0x7f3b0c035d90 comp rx=0 tx=0).stop 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 shutdown_connections 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3afc0a2970 0x7f3afc0a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b1010e010 0x7f3b1019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b100744d0 0x7f3b1019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 --2- 172.21.15.150:0/2870325848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b10073bb0 0x7f3b1019a2f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 >> 172.21.15.150:0/2870325848 conn(0x7f3b10078be0 msgr2=0x7f3b100ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 shutdown_connections 2024-03-18T16:12:02.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.484+0000 7f3b17c54640 1 -- 172.21.15.150:0/2870325848 wait complete. 2024-03-18T16:12:02.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:12:02.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- 172.21.15.150:0/4259419760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60101010 msgr2=0x7f6d6010d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 --2- 172.21.15.150:0/4259419760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60101010 0x7f6d6010d520 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009cd0 tx=0x7f6d5802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- 172.21.15.150:0/4259419760 shutdown_connections 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 --2- 172.21.15.150:0/4259419760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60101010 0x7f6d6010d520 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 --2- 172.21.15.150:0/4259419760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 0x7f6d60100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 --2- 172.21.15.150:0/4259419760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60106660 0x7f6d60106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- 172.21.15.150:0/4259419760 >> 172.21.15.150:0/4259419760 conn(0x7f6d600777e0 msgr2=0x7f6d60077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- 172.21.15.150:0/4259419760 shutdown_connections 2024-03-18T16:12:02.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- 172.21.15.150:0/4259419760 wait complete. 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 Processor -- start 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.600+0000 7f6d662bf640 1 -- start start 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 0x7f6d60198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 0x7f6d60198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60110a70 con 0x7f6d60106660 2024-03-18T16:12:02.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d601108f0 con 0x7f6d60100650 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60110bf0 con 0x7f6d60101010 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52298/0 (socket says 172.21.15.150:52298) 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 -- 172.21.15.150:0/3880572882 learned_addr learned my addr 172.21.15.150:0/3880572882 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5f7fe640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 0x7f6d60198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5ffff640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 0x7f6d60198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 -- 172.21.15.150:0/3880572882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 msgr2=0x7f6d60198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 0x7f6d60198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 -- 172.21.15.150:0/3880572882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 msgr2=0x7f6d60198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 0x7f6d60198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 -- 172.21.15.150:0/3880572882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6019d0c0 con 0x7f6d60106660 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5f7fe640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 0x7f6d60198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d6485c640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6d580348f0 tx=0x7f6d58032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58041020 con 0x7f6d60106660 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6019d350 con 0x7f6d60106660 2024-03-18T16:12:02.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d58032e80 con 0x7f6d60106660 2024-03-18T16:12:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.601+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58039040 con 0x7f6d60106660 2024-03-18T16:12:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.602+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d601a4c40 con 0x7f6d60106660 2024-03-18T16:12:02.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.603+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d600685d0 con 0x7f6d60106660 2024-03-18T16:12:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.604+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f6d58044020 con 0x7f6d60106660 2024-03-18T16:12:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.604+0000 7f6d5d7fa640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 0x7f6d480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.604+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8527+0+0 (secure 0 0 0) 0x7f6d580f0940 con 0x7f6d60106660 2024-03-18T16:12:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.604+0000 7f6d5ffff640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 0x7f6d480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.605+0000 7f6d5ffff640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 0x7f6d480a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009b50 tx=0x7f6d50005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.609+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d5804b020 con 0x7f6d60106660 2024-03-18T16:12:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.804+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6d60109bf0 con 0x7f6d60106660 2024-03-18T16:12:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.804+0000 7f6d5d7fa640 1 -- 172.21.15.150:0/3880572882 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f6d580bd7b0 con 0x7f6d60106660 2024-03-18T16:12:02.807 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T16:12:02.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 msgr2=0x7f6d480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 0x7f6d480a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009b50 tx=0x7f6d50005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 msgr2=0x7f6d6019c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6d580348f0 tx=0x7f6d58032c90 comp rx=0 tx=0).stop 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 shutdown_connections 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6d480a2970 0x7f6d480a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d60106660 0x7f6d6019c990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6d60101010 0x7f6d60198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 --2- 172.21.15.150:0/3880572882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d60100650 0x7f6d60198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 >> 172.21.15.150:0/3880572882 conn(0x7f6d600777e0 msgr2=0x7f6d60068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 shutdown_connections 2024-03-18T16:12:02.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.807+0000 7f6d662bf640 1 -- 172.21.15.150:0/3880572882 wait complete. 2024-03-18T16:12:02.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:02 smithi150 ceph-mon[31946]: from='client.16404 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:02.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:02 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2146873640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:02.819 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:02 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2870325848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:02.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:12:02.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 -- 172.21.15.150:0/3392639106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78100650 msgr2=0x7efd78100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/3392639106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78100650 0x7efd78100ad0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009930 tx=0x7efd6c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 -- 172.21.15.150:0/3392639106 shutdown_connections 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/3392639106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78101010 0x7efd7810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/3392639106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78100650 0x7efd78100ad0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/3392639106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78106660 0x7efd78106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.923+0000 7efd7ebc4640 1 -- 172.21.15.150:0/3392639106 >> 172.21.15.150:0/3392639106 conn(0x7efd780777e0 msgr2=0x7efd78077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- 172.21.15.150:0/3392639106 shutdown_connections 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- 172.21.15.150:0/3392639106 wait complete. 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 Processor -- start 2024-03-18T16:12:02.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- start start 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 0x7efd78198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78106660 0x7efd781a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd78110a70 con 0x7efd78101010 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efd781108f0 con 0x7efd78106660 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7ebc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7efd78110bf0 con 0x7efd78100650 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52316/0 (socket says 172.21.15.150:52316) 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd77fff640 1 -- 172.21.15.150:0/1055192356 learned_addr learned my addr 172.21.15.150:0/1055192356 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7c960640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 0x7efd78198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.924+0000 7efd7d161640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78106660 0x7efd781a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 -- 172.21.15.150:0/1055192356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 msgr2=0x7efd78198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 0x7efd78198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 -- 172.21.15.150:0/1055192356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78106660 msgr2=0x7efd781a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78106660 0x7efd781a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:02.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 -- 172.21.15.150:0/1055192356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd781a44b0 con 0x7efd78101010 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd7c960640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 0x7efd78198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd77fff640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009230 tx=0x7efd6c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c041070 con 0x7efd78101010 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd6c004450 con 0x7efd78101010 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd781a4680 con 0x7efd78101010 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c003e50 con 0x7efd78101010 2024-03-18T16:12:02.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.925+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd781a4ae0 con 0x7efd78101010 2024-03-18T16:12:02.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.926+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd780685d0 con 0x7efd78101010 2024-03-18T16:12:02.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.927+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7efd6c044070 con 0x7efd78101010 2024-03-18T16:12:02.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.928+0000 7efd75ffb640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 0x7efd600a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:02.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.928+0000 7efd7c960640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 0x7efd600a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:02.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.928+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7efd6c0f0640 con 0x7efd78101010 2024-03-18T16:12:02.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.928+0000 7efd7c960640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 0x7efd600a4fd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd68009f40 tx=0x7efd68005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:02.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:02.932+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd6c0bce30 con 0x7efd78101010 2024-03-18T16:12:03.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:02 smithi156 ceph-mon[35919]: from='client.16404 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:03.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:02 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2146873640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:03.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:02 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2870325848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.106+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd78108bf0 con 0x7efd78101010 2024-03-18T16:12:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.107+0000 7efd75ffb640 1 -- 172.21.15.150:0/1055192356 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7efd6c0c0ed0 con 0x7efd78101010 2024-03-18T16:12:03.109 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 msgr2=0x7efd600a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 0x7efd600a4fd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd68009f40 tx=0x7efd68005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 msgr2=0x7efd78199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009230 tx=0x7efd6c02cb30 comp rx=0 tx=0).stop 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 shutdown_connections 2024-03-18T16:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7efd600a2b10 0x7efd600a4fd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd78106660 0x7efd781a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78101010 0x7efd78199430 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 --2- 172.21.15.150:0/1055192356 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd78100650 0x7efd78198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 >> 172.21.15.150:0/1055192356 conn(0x7efd780777e0 msgr2=0x7efd78102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 shutdown_connections 2024-03-18T16:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.109+0000 7efd7ebc4640 1 -- 172.21.15.150:0/1055192356 wait complete. 2024-03-18T16:12:03.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T16:12:03.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:02 smithi191 ceph-mon[36784]: from='client.16404 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:03.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:02 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2146873640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:03.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:02 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2870325848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 -- 172.21.15.150:0/663838715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 msgr2=0x7f5a5010e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/663838715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a5010e630 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009ca0 tx=0x7f5a4802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 -- 172.21.15.150:0/663838715 shutdown_connections 2024-03-18T16:12:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/663838715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a5010e630 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/663838715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a50069e30 0x7f5a5006a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/663838715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a501077e0 0x7f5a50107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 -- 172.21.15.150:0/663838715 >> 172.21.15.150:0/663838715 conn(0x7f5a500764e0 msgr2=0x7f5a500768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 -- 172.21.15.150:0/663838715 shutdown_connections 2024-03-18T16:12:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.222+0000 7f5a54ab2640 1 -- 172.21.15.150:0/663838715 wait complete. 2024-03-18T16:12:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 Processor -- start 2024-03-18T16:12:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 -- start start 2024-03-18T16:12:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50069e30 0x7f5a50198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:03.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501077e0 0x7f5a5019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5a50111bc0 con 0x7f5a5006a7d0 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5a50111a40 con 0x7f5a50069e30 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a54ab2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5a50111d40 con 0x7f5a501077e0 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a4dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a4dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52334/0 (socket says 172.21.15.150:52334) 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a4dd9b640 1 -- 172.21.15.150:0/3434856941 learned_addr learned my addr 172.21.15.150:0/3434856941 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a4e59c640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50069e30 0x7f5a50198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.223+0000 7f5a4ed9d640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501077e0 0x7f5a5019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 -- 172.21.15.150:0/3434856941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501077e0 msgr2=0x7f5a5019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501077e0 0x7f5a5019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 -- 172.21.15.150:0/3434856941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50069e30 msgr2=0x7f5a50198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50069e30 0x7f5a50198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 -- 172.21.15.150:0/3434856941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5019d070 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a4dd9b640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000e9f0 tx=0x7f5a4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000cd80 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a40005500 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5019d360 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40010690 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.224+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a501a4bf0 con 0x7f5a5006a7d0 2024-03-18T16:12:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.225+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a50102ea0 con 0x7f5a5006a7d0 2024-03-18T16:12:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.226+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f5a400046c0 con 0x7f5a5006a7d0 2024-03-18T16:12:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.227+0000 7f5a3f7fe640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 0x7f5a380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.227+0000 7f5a4e59c640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 0x7f5a380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:03.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.227+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8571+0+0 (secure 0 0 0) 0x7f5a400cf470 con 0x7f5a5006a7d0 2024-03-18T16:12:03.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.227+0000 7f5a4e59c640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 0x7f5a380a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009b50 tx=0x7f5a44005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:03.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.230+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a40053740 con 0x7f5a5006a7d0 2024-03-18T16:12:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.450+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f5a50068a20 con 0x7f5a5006a7d0 2024-03-18T16:12:03.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.458+0000 7f5a3f7fe640 1 -- 172.21.15.150:0/3434856941 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740369 (secure 0 0 0) 0x7f5a50068a20 con 0x7f5a5006a7d0 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: "epoch": 25, 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: "active_gid": 15336, 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: "active_name": "x", 2024-03-18T16:12:03.470 INFO:teuthology.orchestra.run.smithi150.stdout: "active_addrs": { 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6800", 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 669533394 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-18T16:12:03.471 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:6801", 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 669533394 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "active_addr": "172.21.15.150:6801/669533394", 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "active_change": "2024-03-18T16:00:04.469098+0000", 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "available": true, 2024-03-18T16:12:03.472 INFO:teuthology.orchestra.run.smithi150.stdout: "standbys": [ 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "gid": 25106, 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "y", 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_features": 4540701547738038271, 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "available_modules": [ 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "alerts", 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.473 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:03.474 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.475 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.476 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.477 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.478 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.479 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.480 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.481 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.482 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_destination": { 2024-03-18T16:12:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_destination", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.484 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_from_name": { 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_from_name", 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.485 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "Ceph", 2024-03-18T16:12:03.486 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.486 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.486 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.487 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email From: name", 2024-03-18T16:12:03.487 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.487 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.487 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_host": { 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_host", 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP server", 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.489 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_password": { 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_password", 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.490 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Password to authenticate with", 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.491 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_port": { 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_port", 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "465", 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.492 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP port", 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_sender": { 2024-03-18T16:12:03.493 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_sender", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.494 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_ssl": { 2024-03-18T16:12:03.495 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_ssl", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.496 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_user": { 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_user", 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "User to authenticate as", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.498 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "balancer", 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.499 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "active": { 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "active", 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.500 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.501 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_time": { 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_time", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0000", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.502 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_weekday": { 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_weekday", 2024-03-18T16:12:03.503 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:12:03.504 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_max_iterations": { 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.505 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "25", 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "250", 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.506 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_metrics": { 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_metrics", 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.507 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.508 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_step": { 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_step", 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.5", 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0.001", 2024-03-18T16:12:03.509 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "0.999", 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.510 INFO:teuthology.orchestra.run.smithi150.stdout: "end_time": { 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_time", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2359", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.511 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "end_weekday": { 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_weekday", 2024-03-18T16:12:03.512 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:12:03.513 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.514 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.515 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.516 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.517 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.518 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.519 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.520 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.521 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.522 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "min_score": { 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_score", 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.524 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:12:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.526 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "upmap", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-compat", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "none", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "read", 2024-03-18T16:12:03.527 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap", 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap-read" 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Balancer mode", 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.528 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_ids": { 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_ids", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.529 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:03.530 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.531 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_deviation": { 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_deviation", 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.532 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_optimizations": { 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.534 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.535 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm", 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.536 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_down_multiplier": { 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_down_multiplier", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.537 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_refresh_rate": { 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_refresh_rate", 2024-03-18T16:12:03.538 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "20", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:12:03.539 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_starting_port": { 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_starting_port", 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.540 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4721", 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:12:03.541 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_ptrace": { 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_ptrace", 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.542 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:12:03.543 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_interval": { 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_interval", 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.544 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.7", 2024-03-18T16:12:03.546 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.547 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm_log_destination": { 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm_log_destination", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.548 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "file", 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "file,syslog", 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "syslog" 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.549 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "cgroups_split": { 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cgroups_split", 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.550 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.551 INFO:teuthology.orchestra.run.smithi150.stdout: "config_checks_enabled": { 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_checks_enabled", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.552 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "config_dashboard": { 2024-03-18T16:12:03.553 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_dashboard", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.554 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_alertmanager": { 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_alertmanager", 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.555 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.556 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_base": { 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_base", 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.557 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.558 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_elasticsearch": { 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:12:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "elasticsearch container image", 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_grafana": { 2024-03-18T16:12:03.560 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_grafana", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.561 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_haproxy": { 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_haproxy", 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.562 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "HAproxy container image", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.563 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_agent": { 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.564 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.565 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_collector": { 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.566 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_query": { 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.568 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger query container image", 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_keepalived": { 2024-03-18T16:12:03.569 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_keepalived", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.570 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Keepalived container image", 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_loki": { 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_loki", 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.571 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Loki container image", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.572 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_node_exporter": { 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_node_exporter", 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.573 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.574 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_nvmeof": { 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_nvmeof", 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:12:03.575 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Nvme-of container image", 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.576 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_prometheus": { 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_prometheus", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.577 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_promtail": { 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_promtail", 2024-03-18T16:12:03.578 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Promtail container image", 2024-03-18T16:12:03.579 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_snmp_gateway": { 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.581 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "container_init": { 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_init", 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.582 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.583 INFO:teuthology.orchestra.run.smithi150.stdout: "daemon_cache_timeout": { 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.584 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:12:03.585 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "900", 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.586 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "default_registry": { 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_registry", 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.587 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.588 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "device_cache_timeout": { 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_cache_timeout", 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.589 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1800", 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.590 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "device_enhanced_scan": { 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_enhanced_scan", 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.591 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "facts_cache_timeout": { 2024-03-18T16:12:03.592 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "facts_cache_timeout", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.593 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "host_check_interval": { 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "host_check_interval", 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.594 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.595 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "hw_monitoring": { 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hw_monitoring", 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "inventory_list_all": { 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "inventory_list_all", 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.599 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.600 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.601 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "log_refresh_metadata": { 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_refresh_metadata", 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.603 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.604 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.605 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.605 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.605 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.606 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.606 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.606 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:12:03.606 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.607 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.608 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.609 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.612 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.613 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "*", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.614 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "max_count_per_host": { 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_count_per_host", 2024-03-18T16:12:03.615 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:12:03.616 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd_draining_count": { 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_osd_draining_count", 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.617 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.618 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "migration_current": { 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "migration_current", 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.619 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "internal - do not modify", 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.620 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "root", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm-package", 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "root" 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "oob_default_addr": { 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "oob_default_addr", 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "169.254.1.1", 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.623 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "prometheus_alerts_path": { 2024-03-18T16:12:03.624 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.625 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_insecure": { 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_insecure", 2024-03-18T16:12:03.626 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:12:03.627 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_password": { 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_password", 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.628 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.629 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_url": { 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_url", 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.630 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.631 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_username": { 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_username", 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.632 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.633 INFO:teuthology.orchestra.run.smithi150.stdout: "secure_monitoring_stack": { 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.634 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "service_discovery_port": { 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "service_discovery_port", 2024-03-18T16:12:03.635 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8765", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:12:03.636 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "ssh_config_file": { 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssh_config_file", 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.637 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.638 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "use_agent": { 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_agent", 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.639 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.640 INFO:teuthology.orchestra.run.smithi150.stdout: "use_repo_digest": { 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_repo_digest", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.641 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_failed_host_check": { 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:12:03.642 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:12:03.643 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_daemons": { 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.644 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.645 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_hosts": { 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.646 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.647 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crash", 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.649 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.650 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.651 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.653 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.654 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.655 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.656 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "retain_interval": { 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retain_interval", 2024-03-18T16:12:03.657 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "31536000", 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.658 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_recent_interval": { 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_recent_interval", 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.659 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.660 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.660 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1209600", 2024-03-18T16:12:03.660 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.660 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.660 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dashboard", 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:12:03.662 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.663 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:12:03.664 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.665 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:12:03.666 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.667 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.668 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.669 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.670 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:12:03.671 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.672 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.673 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.675 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:12:03.676 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.677 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.678 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.679 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.680 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.681 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.682 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.683 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:12:03.684 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.685 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.686 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.687 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.688 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_URL": { 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:12:03.689 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.690 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.691 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:03.692 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.692 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.692 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.692 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:12:03.693 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.694 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:12:03.695 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.696 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.697 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:12:03.698 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.699 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.700 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "{}", 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.701 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.702 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:12:03.703 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.704 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:12:03.705 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.706 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.707 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.708 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.709 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:12:03.710 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.711 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.712 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.713 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.713 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.713 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.713 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:12:03.714 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.715 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.716 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.716 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.716 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.716 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.717 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.718 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.719 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.719 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.719 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:12:03.719 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:12:03.719 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.720 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.721 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.721 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.722 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:12:03.723 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.724 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.725 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8", 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.726 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "45", 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.728 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.729 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.732 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.734 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:12:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.738 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:12:03.739 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.740 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.741 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:12:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:12:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.742 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.743 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.744 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.745 INFO:teuthology.orchestra.run.smithi150.stdout: "cross_origin_url": { 2024-03-18T16:12:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cross_origin_url", 2024-03-18T16:12:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.746 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.747 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "crt_file": { 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crt_file", 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.748 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "debug": { 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "debug", 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.750 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.751 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "jwt_token_ttl": { 2024-03-18T16:12:03.752 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "jwt_token_ttl", 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "28800", 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.753 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.754 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.755 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.756 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.757 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.758 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.759 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.761 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.763 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.764 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.765 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.766 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.767 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.768 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "motd": { 2024-03-18T16:12:03.769 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "motd", 2024-03-18T16:12:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.770 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "The message of the day", 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.772 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.773 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.774 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.775 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.776 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8080", 2024-03-18T16:12:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.778 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.779 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.780 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.781 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl_server_port": { 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl_server_port", 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8443", 2024-03-18T16:12:03.782 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.783 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.784 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "redirect", 2024-03-18T16:12:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.785 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect" 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:03.786 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.787 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "url_prefix": { 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url_prefix", 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.788 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.789 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "devicehealth", 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_monitoring": { 2024-03-18T16:12:03.790 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_monitoring", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "monitor device health metrics", 2024-03-18T16:12:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.792 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.793 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.797 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.798 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.799 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.800 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.801 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "mark_out_threshold": { 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mark_out_threshold", 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2419200", 2024-03-18T16:12:03.802 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.803 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": { 2024-03-18T16:12:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_name", 2024-03-18T16:12:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.804 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "device_health_metrics", 2024-03-18T16:12:03.805 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.806 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "retention_period": { 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retention_period", 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15552000", 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.807 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:12:03.808 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_frequency": { 2024-03-18T16:12:03.809 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_frequency", 2024-03-18T16:12:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.810 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.811 INFO:teuthology.orchestra.run.smithi150.stdout: "self_heal": { 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "self_heal", 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.812 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.813 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:12:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.814 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:03.815 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.816 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_threshold": { 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_threshold", 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.817 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7257600", 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.818 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:12:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.819 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "diskprediction_local", 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.820 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.821 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.822 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.823 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.824 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.825 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.826 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.827 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.828 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.829 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "predict_interval": { 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predict_interval", 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.830 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.831 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.832 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "predictor_model": { 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predictor_model", 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "prophetstor", 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.834 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.835 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.836 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.838 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "influx", 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": false, 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "batch_size": { 2024-03-18T16:12:03.839 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "batch_size", 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5000", 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.840 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "database": { 2024-03-18T16:12:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "database", 2024-03-18T16:12:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.842 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "ceph", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.843 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": { 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hostname", 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.844 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.845 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "30", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "5", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.847 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.848 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.849 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.850 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.852 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.852 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.852 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.852 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.852 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.853 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.854 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.854 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.854 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.855 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.856 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.857 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.857 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.857 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.857 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.858 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.858 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.858 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "password": { 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "password", 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.859 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.860 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.860 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.860 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:12:03.860 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.860 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.861 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.861 INFO:teuthology.orchestra.run.smithi150.stdout: "port": { 2024-03-18T16:12:03.861 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "port", 2024-03-18T16:12:03.861 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8086", 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.862 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server port", 2024-03-18T16:12:03.863 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.863 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.863 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.864 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "false", 2024-03-18T16:12:03.865 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.865 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.865 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.865 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:03.865 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "threads": { 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threads", 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.866 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.867 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:03.867 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:03.867 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "32", 2024-03-18T16:12:03.867 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.867 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "username": { 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "username", 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.868 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.869 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: "verify_ssl": { 2024-03-18T16:12:03.870 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "verify_ssl", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "true", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.872 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "insights", 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.873 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.874 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.875 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.876 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.877 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.878 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.879 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.880 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "iostat", 2024-03-18T16:12:03.881 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.882 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.883 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.884 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.885 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.886 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.887 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.888 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.889 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.890 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.891 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "k8sevents", 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_event_retention_days": { 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.892 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "config_check_secs": { 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_check_secs", 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "10", 2024-03-18T16:12:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:12:03.897 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.898 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.899 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.900 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.901 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.902 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.903 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.904 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.905 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.906 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.907 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "localpool", 2024-03-18T16:12:03.908 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.909 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.909 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.909 INFO:teuthology.orchestra.run.smithi150.stdout: "failure_domain": { 2024-03-18T16:12:03.909 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "failure_domain", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "host", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.910 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.911 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.912 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.913 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.914 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.915 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.916 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.917 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.918 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": { 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_size", 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.919 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "num_rep": { 2024-03-18T16:12:03.920 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "num_rep", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.921 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": { 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_num", 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:03.922 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "128", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.923 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "prefix": { 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prefix", 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.924 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "subtree": { 2024-03-18T16:12:03.925 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "subtree", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "rack", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.926 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mds_autoscaler", 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.927 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.928 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.929 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.930 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.931 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.932 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.933 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.934 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.935 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirroring", 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.936 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.937 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.938 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.939 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.940 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.941 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.942 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.943 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "nfs", 2024-03-18T16:12:03.944 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.945 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.946 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.949 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.950 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.951 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.952 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "fail_fs": { 2024-03-18T16:12:03.953 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "fail_fs", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.954 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.955 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.956 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.957 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.958 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.960 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.961 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.962 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator": { 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.963 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm", 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "rook", 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "test_orchestrator" 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Orchestrator backend", 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.964 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_perf_query", 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.965 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.966 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.967 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.968 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.969 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.970 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.971 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.972 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.973 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_support", 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.974 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.975 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.976 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.977 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.978 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.979 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.981 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_autoscaler", 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.982 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.983 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.984 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.985 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:03.986 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.987 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.988 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:03.990 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.991 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "threshold": { 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threshold", 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:03.992 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1.0", 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "scaling threshold", 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.993 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "progress", 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_pg_recovery_event": { 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.994 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.995 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.996 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:03.997 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:03.998 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:03.999 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.000 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.001 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.002 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.003 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.004 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "max_completed_events": { 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_completed_events", 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.005 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "50", 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.007 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.009 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus", 2024-03-18T16:12:04.009 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "cache": { 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cache", 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.010 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "exclude_perf_counters": { 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "exclude_perf_counters", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.013 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.014 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.015 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.016 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.017 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.018 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.019 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.020 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools": { 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools", 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.021 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.022 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "300", 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.023 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_interval": { 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_interval", 2024-03-18T16:12:04.024 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15.0", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:04.026 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:04.027 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "9283", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.028 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "stale_cache_strategy": { 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stale_cache_strategy", 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.029 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "log", 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.030 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "default", 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.031 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "default", 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "error" 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.032 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:04.033 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.034 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_support", 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.036 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.037 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.038 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.039 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.040 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.041 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_snap_create": { 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.043 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.044 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.045 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "trash_purge_schedule": { 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "trash_purge_schedule", 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.046 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.047 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "restful", 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_auth": { 2024-03-18T16:12:04.048 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_auth", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.049 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.050 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.051 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.052 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.053 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.054 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.055 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.056 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.057 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.058 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.060 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.061 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.062 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rgw", 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.063 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.064 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.065 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.066 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.068 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.069 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.070 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.071 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rook", 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.072 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.073 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.074 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.075 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.076 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.077 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.078 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.079 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "storage_class": { 2024-03-18T16:12:04.080 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "storage_class", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "local", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:12:04.081 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "selftest", 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.082 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.083 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.084 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.085 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.086 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.088 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.089 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "roption1": { 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption1", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.091 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "roption2": { 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption2", 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.092 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "xyz", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.093 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption1": { 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption1", 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.094 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption2": { 2024-03-18T16:12:04.095 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption2", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.096 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption3": { 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption3", 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.097 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.098 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.101 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption4": { 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption4", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.102 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption5": { 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption5", 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.103 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption6": { 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption6", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption7": { 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption7", 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.106 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "42", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.107 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "testkey": { 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testkey", 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.108 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "testlkey": { 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testlkey", 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.109 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.110 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "testnewline": { 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testnewline", 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.111 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.112 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snap_schedule", 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_m_granularity": { 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_m_granularity", 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.113 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "dump_on_update": { 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dump_on_update", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.115 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.116 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.117 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.118 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.119 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.120 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.121 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.122 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.123 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.124 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stats", 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.126 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.128 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.129 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.130 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.131 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "status", 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.133 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.134 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.135 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.136 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.137 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.138 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.139 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.140 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.141 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telegraf", 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "address": { 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "address", 2024-03-18T16:12:04.142 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.143 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.144 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15", 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.145 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.146 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.147 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.148 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.149 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.150 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.151 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.152 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.153 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telemetry", 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_basic": { 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_basic", 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.154 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_crash": { 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_crash", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.156 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_device": { 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_device", 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.157 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.158 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_ident": { 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_ident", 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.159 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_perf": { 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_perf", 2024-03-18T16:12:04.160 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.161 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "contact": { 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "contact", 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.162 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "description": { 2024-03-18T16:12:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "description", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.164 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "device_url": { 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_url", 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.165 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.166 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.167 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.168 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "24", 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "8", 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.169 INFO:teuthology.orchestra.run.smithi150.stdout: "last_opt_revision": { 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "last_opt_revision", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.170 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard": { 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard", 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.171 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard_description": { 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard_description", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.174 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.175 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.177 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.179 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.180 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.181 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "organization": { 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "organization", 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.182 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "proxy": { 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "proxy", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.184 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "url": { 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url", 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.185 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.186 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "test_orchestrator", 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.187 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.188 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.189 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.190 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.192 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.193 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.194 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.195 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "volumes", 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.196 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.197 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.198 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.199 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.200 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.201 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.202 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.203 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_clones": { 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_clones", 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4", 2024-03-18T16:12:04.204 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "periodic_async_work": { 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "periodic_async_work", 2024-03-18T16:12:04.205 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Periodically check for async work", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.206 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_delay": { 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.207 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.208 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.209 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix", 2024-03-18T16:12:04.210 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "discovery_interval": { 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "discovery_interval", 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "100", 2024-03-18T16:12:04.211 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "identifier": { 2024-03-18T16:12:04.212 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "identifier", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.213 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.214 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.215 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.216 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.217 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.219 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.220 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.222 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_host": { 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_host", 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.223 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.224 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_port": { 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_port", 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10051", 2024-03-18T16:12:04.225 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_sender": { 2024-03-18T16:12:04.226 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_sender", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.227 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: "gid": 25159, 2024-03-18T16:12:04.228 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "z", 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr_features": 4540701547738038271, 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "available_modules": [ 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "alerts", 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.229 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.230 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.231 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.232 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.233 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.234 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.236 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.237 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.238 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_destination": { 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_destination", 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.239 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_from_name": { 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_from_name", 2024-03-18T16:12:04.240 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "Ceph", 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.241 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email From: name", 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_host": { 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_host", 2024-03-18T16:12:04.242 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.243 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP server", 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_password": { 2024-03-18T16:12:04.244 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_password", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Password to authenticate with", 2024-03-18T16:12:04.245 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_port": { 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_port", 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.246 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "465", 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP port", 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.247 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_sender": { 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_sender", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_ssl": { 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_ssl", 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.249 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.250 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_user": { 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_user", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.251 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "User to authenticate as", 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.252 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "balancer", 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "active": { 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "active", 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.253 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.254 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_time": { 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_time", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0000", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.255 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_weekday": { 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_weekday", 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.256 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.257 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_max_iterations": { 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "25", 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.258 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "250", 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_metrics": { 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_metrics", 2024-03-18T16:12:04.259 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:12:04.260 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_step": { 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_step", 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.5", 2024-03-18T16:12:04.261 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0.001", 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "0.999", 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "end_time": { 2024-03-18T16:12:04.262 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_time", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2359", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:12:04.263 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "end_weekday": { 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_weekday", 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.264 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.265 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.266 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.267 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.268 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.270 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.271 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.272 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "min_score": { 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_score", 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.273 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.274 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "upmap", 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.275 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-compat", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "none", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "read", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap-read" 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Balancer mode", 2024-03-18T16:12:04.276 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_ids": { 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_ids", 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.277 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.278 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.279 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_deviation": { 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_deviation", 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.280 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.281 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_optimizations": { 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.282 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm", 2024-03-18T16:12:04.283 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_down_multiplier": { 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_down_multiplier", 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:04.284 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_refresh_rate": { 2024-03-18T16:12:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_refresh_rate", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "20", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:12:04.286 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_starting_port": { 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_starting_port", 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.287 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4721", 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.288 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_ptrace": { 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_ptrace", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.289 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_interval": { 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_interval", 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.290 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.291 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.7", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.292 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm_log_destination": { 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm_log_destination", 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.293 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "file", 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "file,syslog", 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "syslog" 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.294 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "cgroups_split": { 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cgroups_split", 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.295 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.296 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "config_checks_enabled": { 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_checks_enabled", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.297 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "config_dashboard": { 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_dashboard", 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.298 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.299 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_alertmanager": { 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_alertmanager", 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:12:04.300 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_base": { 2024-03-18T16:12:04.301 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_base", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:12:04.302 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_elasticsearch": { 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:12:04.303 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.304 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.305 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.306 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:12:04.306 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.306 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.306 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "elasticsearch container image", 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.307 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_grafana": { 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_grafana", 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.308 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.309 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.309 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.309 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.309 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_haproxy": { 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_haproxy", 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.310 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "HAproxy container image", 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_agent": { 2024-03-18T16:12:04.311 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.312 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_collector": { 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.313 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.314 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_query": { 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.315 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger query container image", 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_keepalived": { 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_keepalived", 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.316 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Keepalived container image", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.317 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_loki": { 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_loki", 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.318 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Loki container image", 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_node_exporter": { 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_node_exporter", 2024-03-18T16:12:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.320 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_nvmeof": { 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_nvmeof", 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:12:04.321 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Nvme-of container image", 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_prometheus": { 2024-03-18T16:12:04.322 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_prometheus", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.323 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_promtail": { 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_promtail", 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.324 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Promtail container image", 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_snmp_gateway": { 2024-03-18T16:12:04.325 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:12:04.326 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "container_init": { 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_init", 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.327 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.328 INFO:teuthology.orchestra.run.smithi150.stdout: "daemon_cache_timeout": { 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.329 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.330 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "900", 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "default_registry": { 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_registry", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.332 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "device_cache_timeout": { 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_cache_timeout", 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.333 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1800", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.334 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "device_enhanced_scan": { 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_enhanced_scan", 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.335 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "facts_cache_timeout": { 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "facts_cache_timeout", 2024-03-18T16:12:04.336 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.337 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "host_check_interval": { 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "host_check_interval", 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.338 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "hw_monitoring": { 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hw_monitoring", 2024-03-18T16:12:04.339 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.340 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "inventory_list_all": { 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "inventory_list_all", 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.341 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.342 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.343 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.344 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "log_refresh_metadata": { 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_refresh_metadata", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.345 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.346 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.348 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.349 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.350 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.351 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.352 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:12:04.353 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "*", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:12:04.354 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "max_count_per_host": { 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_count_per_host", 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.355 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.356 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd_draining_count": { 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_osd_draining_count", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.357 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "migration_current": { 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "migration_current", 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.358 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "internal - do not modify", 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.359 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "root", 2024-03-18T16:12:04.360 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm-package", 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "root" 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:12:04.361 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "oob_default_addr": { 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "oob_default_addr", 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.362 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "169.254.1.1", 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.363 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "prometheus_alerts_path": { 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.364 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_insecure": { 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_insecure", 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.365 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.366 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_password": { 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_password", 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.367 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_url": { 2024-03-18T16:12:04.368 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_url", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:12:04.369 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_username": { 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_username", 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.370 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.371 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.372 INFO:teuthology.orchestra.run.smithi150.stdout: "secure_monitoring_stack": { 2024-03-18T16:12:04.372 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:12:04.372 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.372 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.373 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "service_discovery_port": { 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "service_discovery_port", 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.374 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8765", 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.375 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "ssh_config_file": { 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssh_config_file", 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.376 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "use_agent": { 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_agent", 2024-03-18T16:12:04.377 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.378 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "use_repo_digest": { 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_repo_digest", 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.379 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.380 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_failed_host_check": { 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.381 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_daemons": { 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.382 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.383 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_hosts": { 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.384 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.385 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crash", 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.386 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.387 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.388 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.389 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.390 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.391 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.392 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.393 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "retain_interval": { 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retain_interval", 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.394 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "31536000", 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.395 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_recent_interval": { 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_recent_interval", 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1209600", 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.396 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.397 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dashboard", 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.398 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.401 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.402 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.403 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.404 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.405 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.406 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:12:04.407 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.408 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.409 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:12:04.410 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.411 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.412 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.413 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.414 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.415 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.416 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.417 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:12:04.418 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.419 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:04.420 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:12:04.421 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.422 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_URL": { 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.423 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.425 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.426 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.427 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.428 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.429 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.431 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:12:04.432 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "{}", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.433 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.434 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.435 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.436 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.437 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.439 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.440 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.441 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.442 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:12:04.443 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.444 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.445 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.446 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.447 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.448 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.451 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.452 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8", 2024-03-18T16:12:04.453 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:12:04.454 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "45", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.455 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.456 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.458 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.459 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.460 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.461 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:12:04.462 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.463 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:12:04.465 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.466 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.467 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "cross_origin_url": { 2024-03-18T16:12:04.468 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cross_origin_url", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.469 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "crt_file": { 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crt_file", 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.470 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.471 INFO:teuthology.orchestra.run.smithi150.stdout: "debug": { 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "debug", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.472 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "jwt_token_ttl": { 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "jwt_token_ttl", 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.473 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "28800", 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.475 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.476 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.477 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.479 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.480 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.481 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.482 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.483 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "motd": { 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "motd", 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.484 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "The message of the day", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.486 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:04.487 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.488 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8080", 2024-03-18T16:12:04.489 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.490 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.491 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl_server_port": { 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl_server_port", 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.492 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8443", 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.493 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "redirect", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect" 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.495 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.496 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "url_prefix": { 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url_prefix", 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.497 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.498 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "devicehealth", 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_monitoring": { 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_monitoring", 2024-03-18T16:12:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "monitor device health metrics", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.500 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.501 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.502 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.503 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.504 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.505 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.506 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.507 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.509 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "mark_out_threshold": { 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mark_out_threshold", 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.510 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2419200", 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.511 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": { 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_name", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "device_health_metrics", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.512 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "retention_period": { 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retention_period", 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.513 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15552000", 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.514 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_frequency": { 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_frequency", 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "self_heal": { 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "self_heal", 2024-03-18T16:12:04.516 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.517 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.518 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.519 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_threshold": { 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_threshold", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7257600", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.520 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.521 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "diskprediction_local", 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.522 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.523 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.523 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.523 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.523 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.524 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.524 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.524 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.524 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.524 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.525 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.526 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.527 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.528 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.529 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "predict_interval": { 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predict_interval", 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.531 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "predictor_model": { 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predictor_model", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "prophetstor", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.533 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.534 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.535 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "influx", 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": false, 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.536 INFO:teuthology.orchestra.run.smithi150.stdout: "batch_size": { 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "batch_size", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5000", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.537 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "database": { 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "database", 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.538 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "ceph", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": { 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hostname", 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.540 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.541 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "30", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "5", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:12:04.542 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.543 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.544 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.545 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.546 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.548 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.549 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.550 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "password": { 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "password", 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.551 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "port": { 2024-03-18T16:12:04.552 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "port", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8086", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server port", 2024-03-18T16:12:04.553 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.554 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "false", 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.555 INFO:teuthology.orchestra.run.smithi150.stdout: "threads": { 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threads", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "32", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:12:04.556 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "username": { 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "username", 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.557 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.558 INFO:teuthology.orchestra.run.smithi150.stdout: "verify_ssl": { 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "verify_ssl", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "true", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.559 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "insights", 2024-03-18T16:12:04.560 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.561 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.562 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.563 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.564 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.565 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.566 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.567 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.568 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "iostat", 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.569 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.570 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.571 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.572 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.573 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.574 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.575 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.576 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "k8sevents", 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_event_retention_days": { 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:12:04.578 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.579 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "config_check_secs": { 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_check_secs", 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "10", 2024-03-18T16:12:04.580 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.581 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.582 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.583 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.584 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.585 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.586 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.587 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.588 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.589 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "localpool", 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "failure_domain": { 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "failure_domain", 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.590 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "host", 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.591 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.592 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.593 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.594 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.595 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.596 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.597 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.598 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": { 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_size", 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.600 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "num_rep": { 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "num_rep", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.601 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": { 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_num", 2024-03-18T16:12:04.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "128", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.603 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "prefix": { 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prefix", 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.604 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.605 INFO:teuthology.orchestra.run.smithi150.stdout: "subtree": { 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "subtree", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "rack", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.606 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mds_autoscaler", 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.607 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.608 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.609 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.610 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.611 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.612 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.613 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.614 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.615 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirroring", 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.616 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.617 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.618 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.619 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.620 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.621 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "nfs", 2024-03-18T16:12:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.625 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.626 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.627 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.628 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.629 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.630 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.631 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.632 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.633 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "fail_fs": { 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "fail_fs", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.634 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.635 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.636 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.637 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.639 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.640 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.642 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator": { 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.643 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "rook", 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: "test_orchestrator" 2024-03-18T16:12:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Orchestrator backend", 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.645 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_perf_query", 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.646 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.647 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.648 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.649 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.650 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.651 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.652 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.653 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_support", 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.654 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.655 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.656 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.658 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.659 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.660 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.661 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.662 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_autoscaler", 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.663 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.664 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.665 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.666 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.667 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.668 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.669 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.670 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.672 INFO:teuthology.orchestra.run.smithi150.stdout: "threshold": { 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threshold", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1.0", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.673 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "scaling threshold", 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "progress", 2024-03-18T16:12:04.674 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_pg_recovery_event": { 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.675 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:04.676 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.677 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.678 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.679 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.680 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.681 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.682 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.683 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.684 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.685 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "max_completed_events": { 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_completed_events", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "50", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.686 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.687 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.688 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus", 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.689 INFO:teuthology.orchestra.run.smithi150.stdout: "cache": { 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cache", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.690 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "exclude_perf_counters": { 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "exclude_perf_counters", 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.691 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.692 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.693 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.694 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.695 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.697 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.698 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.699 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.700 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools": { 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.702 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "300", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_interval": { 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_interval", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15.0", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.704 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.705 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.706 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "9283", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.707 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "stale_cache_strategy": { 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stale_cache_strategy", 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.708 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "log", 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.709 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "default", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.710 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "default", 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "error" 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.711 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.713 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_support", 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.715 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.716 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.717 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.718 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.719 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.720 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_snap_create": { 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.722 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:12:04.723 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.724 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "trash_purge_schedule": { 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "trash_purge_schedule", 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.725 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.726 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "restful", 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_auth": { 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_auth", 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.727 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.728 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.729 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.730 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.731 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.732 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.733 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.734 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.735 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.737 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:04.738 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.739 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.740 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.741 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rgw", 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.742 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.743 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.744 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.745 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.746 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.747 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.748 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.749 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.750 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rook", 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.751 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.752 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.753 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.754 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.755 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.756 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.757 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.758 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "storage_class": { 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "storage_class", 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.759 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "local", 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.760 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "selftest", 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.761 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.762 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.763 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.764 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.767 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.768 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "roption1": { 2024-03-18T16:12:04.769 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption1", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.770 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "roption2": { 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption2", 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.771 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "xyz", 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.772 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption1": { 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption1", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.773 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption2": { 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption2", 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.774 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.775 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption3": { 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption3", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.776 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption4": { 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption4", 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.777 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.778 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption5": { 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption5", 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption6": { 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption6", 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.780 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.781 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption7": { 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption7", 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.782 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "42", 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "testkey": { 2024-03-18T16:12:04.783 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testkey", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.784 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "testlkey": { 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testlkey", 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.785 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.786 INFO:teuthology.orchestra.run.smithi150.stdout: "testnewline": { 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testnewline", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.787 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snap_schedule", 2024-03-18T16:12:04.788 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_m_granularity": { 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_m_granularity", 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.789 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "dump_on_update": { 2024-03-18T16:12:04.790 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dump_on_update", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:12:04.791 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.792 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.793 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.794 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.795 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.796 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.797 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.798 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.799 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stats", 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.800 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.801 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.802 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.803 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.804 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.805 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.806 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.807 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "status", 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.809 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.810 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.811 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.812 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.813 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.814 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.815 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.816 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telegraf", 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.817 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "address": { 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "address", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.818 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.819 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.820 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.821 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.822 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.824 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.825 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.826 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.827 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.828 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telemetry", 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.829 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_basic": { 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_basic", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.830 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_crash": { 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_crash", 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.831 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.832 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_device": { 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_device", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.833 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_ident": { 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_ident", 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.834 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.835 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_perf": { 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_perf", 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.836 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "contact": { 2024-03-18T16:12:04.837 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "contact", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "description": { 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "description", 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.839 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.840 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "device_url": { 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_url", 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.841 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:04.842 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.844 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "24", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "8", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.845 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "last_opt_revision": { 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "last_opt_revision", 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1", 2024-03-18T16:12:04.846 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard": { 2024-03-18T16:12:04.847 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.848 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard_description": { 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard_description", 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.849 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.851 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.852 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.853 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.854 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.856 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.857 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.858 INFO:teuthology.orchestra.run.smithi150.stdout: "organization": { 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "organization", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.859 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "proxy": { 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "proxy", 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.860 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "url": { 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url", 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.862 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.863 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "test_orchestrator", 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.864 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.865 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.866 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.867 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.868 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.869 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.870 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.871 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "volumes", 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.872 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.873 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.874 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.875 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.876 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.877 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.878 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.879 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_clones": { 2024-03-18T16:12:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_clones", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.881 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "periodic_async_work": { 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "periodic_async_work", 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.882 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Periodically check for async work", 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_delay": { 2024-03-18T16:12:04.883 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.884 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.885 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.886 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix", 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "discovery_interval": { 2024-03-18T16:12:04.887 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "discovery_interval", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "100", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.888 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "identifier": { 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "identifier", 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.889 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.890 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.891 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.891 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.891 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.891 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.891 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.892 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.893 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.894 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.894 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.894 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.894 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.894 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.895 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.895 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.895 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.896 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.896 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.896 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.897 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.897 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.897 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.897 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.898 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.899 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.900 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.901 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.902 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.903 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_host": { 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_host", 2024-03-18T16:12:04.904 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.905 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_port": { 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_port", 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10051", 2024-03-18T16:12:04.906 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_sender": { 2024-03-18T16:12:04.907 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_sender", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.908 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.909 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "modules": [ 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm", 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "dashboard", 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "iostat", 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "nfs", 2024-03-18T16:12:04.910 INFO:teuthology.orchestra.run.smithi150.stdout: "restful", 2024-03-18T16:12:04.911 INFO:teuthology.orchestra.run.smithi150.stdout: "snap_schedule" 2024-03-18T16:12:04.911 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.911 INFO:teuthology.orchestra.run.smithi150.stdout: "available_modules": [ 2024-03-18T16:12:04.911 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "alerts", 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.912 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.914 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.915 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.916 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.916 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.916 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.916 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.917 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.917 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.917 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.917 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.917 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.918 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.919 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.920 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.921 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.922 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.923 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_destination": { 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_destination", 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.924 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_from_name": { 2024-03-18T16:12:04.925 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_from_name", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "Ceph", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Email From: name", 2024-03-18T16:12:04.926 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_host": { 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_host", 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.927 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP server", 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.928 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_password": { 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_password", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.929 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Password to authenticate with", 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_port": { 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_port", 2024-03-18T16:12:04.930 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "465", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP port", 2024-03-18T16:12:04.931 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_sender": { 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_sender", 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.932 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.933 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_ssl": { 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_ssl", 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.934 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "smtp_user": { 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "smtp_user", 2024-03-18T16:12:04.935 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "User to authenticate as", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.936 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "balancer", 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.937 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "active": { 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "active", 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.938 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_time": { 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_time", 2024-03-18T16:12:04.939 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0000", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:04.940 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "begin_weekday": { 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "begin_weekday", 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.941 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.942 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_max_iterations": { 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "25", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "250", 2024-03-18T16:12:04.943 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_metrics": { 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_metrics", 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:12:04.945 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "crush_compat_step": { 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crush_compat_step", 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.5", 2024-03-18T16:12:04.946 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0.001", 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "0.999", 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "end_time": { 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_time", 2024-03-18T16:12:04.947 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2359", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:12:04.948 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "end_weekday": { 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "end_weekday", 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.949 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "0", 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "6", 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:04.950 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.951 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.952 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.953 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.954 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:04.955 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:04.956 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.957 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "min_score": { 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_score", 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:04.958 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:04.959 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "upmap", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "crush-compat", 2024-03-18T16:12:04.960 INFO:teuthology.orchestra.run.smithi150.stdout: "none", 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "read", 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap", 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap-read" 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Balancer mode", 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.961 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_ids": { 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_ids", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.962 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.963 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.964 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_deviation": { 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_deviation", 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:04.965 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: "upmap_max_optimizations": { 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:04.968 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm", 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_down_multiplier": { 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_down_multiplier", 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.969 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.970 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_refresh_rate": { 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_refresh_rate", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "20", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.971 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "agent_starting_port": { 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "agent_starting_port", 2024-03-18T16:12:04.972 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4721", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:12:04.973 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_ptrace": { 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_ptrace", 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.974 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.975 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_interval": { 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_interval", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.976 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.977 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0.7", 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.978 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm_log_destination": { 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cephadm_log_destination", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.979 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "file", 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "file,syslog", 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "syslog" 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.980 INFO:teuthology.orchestra.run.smithi150.stdout: "cgroups_split": { 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cgroups_split", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.981 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "config_checks_enabled": { 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_checks_enabled", 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.982 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.983 INFO:teuthology.orchestra.run.smithi150.stdout: "config_dashboard": { 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_dashboard", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.984 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_alertmanager": { 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_alertmanager", 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.985 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.986 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_base": { 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_base", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.987 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_elasticsearch": { 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.988 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "elasticsearch container image", 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.989 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_grafana": { 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_grafana", 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.990 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_haproxy": { 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_haproxy", 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.991 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "HAproxy container image", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.992 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_agent": { 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.993 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_collector": { 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.994 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.995 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_jaeger_query": { 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.996 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Jaeger query container image", 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_keepalived": { 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_keepalived", 2024-03-18T16:12:04.997 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Keepalived container image", 2024-03-18T16:12:04.998 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_loki": { 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_loki", 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:04.999 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Loki container image", 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.000 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_node_exporter": { 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_node_exporter", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.001 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_nvmeof": { 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_nvmeof", 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.002 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Nvme-of container image", 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.003 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_prometheus": { 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_prometheus", 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.004 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Prometheus container image", 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_promtail": { 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_promtail", 2024-03-18T16:12:05.005 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Promtail container image", 2024-03-18T16:12:05.006 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "container_image_snmp_gateway": { 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.007 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "container_init": { 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "container_init", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.009 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "daemon_cache_timeout": { 2024-03-18T16:12:05.010 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:12:05.011 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.012 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "900", 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.013 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "default_registry": { 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "default_registry", 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "docker.io", 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.014 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "device_cache_timeout": { 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_cache_timeout", 2024-03-18T16:12:05.015 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1800", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.016 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "device_enhanced_scan": { 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_enhanced_scan", 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.017 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "facts_cache_timeout": { 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "facts_cache_timeout", 2024-03-18T16:12:05.018 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:12:05.019 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "host_check_interval": { 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "host_check_interval", 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.020 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.021 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "hw_monitoring": { 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hw_monitoring", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.022 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "inventory_list_all": { 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "inventory_list_all", 2024-03-18T16:12:05.023 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.024 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.025 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.026 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "log_refresh_metadata": { 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_refresh_metadata", 2024-03-18T16:12:05.027 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.028 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.029 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.031 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.032 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.033 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.034 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.035 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:12:05.036 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "*", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:12:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "max_count_per_host": { 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_count_per_host", 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.038 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.039 INFO:teuthology.orchestra.run.smithi150.stdout: "max_osd_draining_count": { 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_osd_draining_count", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.040 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "migration_current": { 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "migration_current", 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.041 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "internal - do not modify", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.042 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "mode": { 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mode", 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "root", 2024-03-18T16:12:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm-package", 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "root" 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.044 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "oob_default_addr": { 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "oob_default_addr", 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "169.254.1.1", 2024-03-18T16:12:05.045 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.046 INFO:teuthology.orchestra.run.smithi150.stdout: "prometheus_alerts_path": { 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.047 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_insecure": { 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_insecure", 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.048 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.049 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_password": { 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_password", 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.050 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_url": { 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_url", 2024-03-18T16:12:05.051 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.052 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "registry_username": { 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "registry_username", 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.053 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "secure_monitoring_stack": { 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:12:05.054 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.055 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "service_discovery_port": { 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "service_discovery_port", 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8765", 2024-03-18T16:12:05.056 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "ssh_config_file": { 2024-03-18T16:12:05.057 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssh_config_file", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:12:05.058 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "use_agent": { 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_agent", 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.059 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.060 INFO:teuthology.orchestra.run.smithi150.stdout: "use_repo_digest": { 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "use_repo_digest", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.061 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_failed_host_check": { 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.062 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.063 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_daemons": { 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.064 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_on_stray_hosts": { 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:12:05.065 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.066 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crash", 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.067 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.068 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.069 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.070 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.071 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.072 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.073 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.074 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "retain_interval": { 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retain_interval", 2024-03-18T16:12:05.075 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "31536000", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.076 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_recent_interval": { 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_recent_interval", 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1209600", 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.077 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.078 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dashboard", 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.079 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.080 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.081 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.082 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.083 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.084 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.085 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.086 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.087 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:12:05.088 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.089 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.090 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.092 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:12:05.093 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.094 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.095 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:12:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.097 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.098 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:12:05.099 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.100 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:05.101 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:12:05.102 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.103 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_URL": { 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.104 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.105 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:05.106 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:12:05.107 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.108 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.109 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.110 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:12:05.112 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.113 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "{}", 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.115 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.116 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:12:05.117 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.118 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.119 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.120 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.121 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:12:05.122 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.123 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.124 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.126 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.127 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.128 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.129 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:12:05.130 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.131 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.132 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:12:05.133 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.134 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.135 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.136 INFO:teuthology.orchestra.run.smithi150.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.137 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.138 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "45", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.139 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.140 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:12:05.141 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "admin", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.142 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.143 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.144 INFO:teuthology.orchestra.run.smithi150.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.145 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.146 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:12:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.150 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.151 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.152 INFO:teuthology.orchestra.run.smithi150.stdout: "cross_origin_url": { 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cross_origin_url", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.153 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "crt_file": { 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "crt_file", 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.154 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.155 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "debug": { 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "debug", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.156 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "jwt_token_ttl": { 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "jwt_token_ttl", 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.157 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "28800", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.158 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.159 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.161 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.162 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.163 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.164 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.165 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.166 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.167 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "motd": { 2024-03-18T16:12:05.168 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "motd", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "The message of the day", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.169 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.170 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.171 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.172 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.173 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8080", 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.174 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.175 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl_server_port": { 2024-03-18T16:12:05.176 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl_server_port", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8443", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.177 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.178 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "redirect", 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "redirect" 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.179 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:05.180 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "url_prefix": { 2024-03-18T16:12:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url_prefix", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.182 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "devicehealth", 2024-03-18T16:12:05.183 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_monitoring": { 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_monitoring", 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.184 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "monitor device health metrics", 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.185 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.186 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.187 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.188 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.189 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.190 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.191 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.192 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.193 INFO:teuthology.orchestra.run.smithi150.stdout: "mark_out_threshold": { 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mark_out_threshold", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "2419200", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.194 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "pool_name": { 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pool_name", 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.195 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "device_health_metrics", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.196 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "retention_period": { 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "retention_period", 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15552000", 2024-03-18T16:12:05.197 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_frequency": { 2024-03-18T16:12:05.198 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_frequency", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:12:05.199 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "self_heal": { 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "self_heal", 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.200 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.201 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.202 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "warn_threshold": { 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "warn_threshold", 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.203 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7257600", 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.204 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "diskprediction_local", 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.205 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.206 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.207 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.208 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.209 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.210 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.211 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.212 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "predict_interval": { 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predict_interval", 2024-03-18T16:12:05.213 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "86400", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.214 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "predictor_model": { 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "predictor_model", 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.215 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "prophetstor", 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.216 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "600", 2024-03-18T16:12:05.217 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.218 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "influx", 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": false, 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "batch_size": { 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "batch_size", 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.219 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5000", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.220 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "database": { 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "database", 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "ceph", 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.221 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": { 2024-03-18T16:12:05.222 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "hostname", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:12:05.223 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.224 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "30", 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "5", 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.226 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.227 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.228 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.229 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.230 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.231 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.232 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.233 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "password": { 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "password", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.234 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "port": { 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "port", 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.235 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "8086", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "InfluxDB server port", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.236 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "ssl": { 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ssl", 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "false", 2024-03-18T16:12:05.237 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "threads": { 2024-03-18T16:12:05.238 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threads", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "32", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:12:05.239 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "username": { 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "username", 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.241 INFO:teuthology.orchestra.run.smithi150.stdout: "verify_ssl": { 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "verify_ssl", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "true", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.242 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.243 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "insights", 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.244 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.245 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.246 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.247 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.248 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.249 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.250 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.251 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "iostat", 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.252 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.253 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.254 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.255 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.257 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.259 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "k8sevents", 2024-03-18T16:12:05.260 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_event_retention_days": { 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "7", 2024-03-18T16:12:05.261 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.262 INFO:teuthology.orchestra.run.smithi150.stdout: "config_check_secs": { 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "config_check_secs", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "10", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.263 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.264 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.265 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.267 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.269 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.271 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "localpool", 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.272 INFO:teuthology.orchestra.run.smithi150.stdout: "failure_domain": { 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "failure_domain", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "host", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.273 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.274 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.275 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.276 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.277 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.278 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.279 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.280 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.281 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "min_size": { 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "min_size", 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.282 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.283 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "num_rep": { 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "num_rep", 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3", 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.284 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_num": { 2024-03-18T16:12:05.285 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_num", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "128", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.286 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "prefix": { 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prefix", 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.287 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "subtree": { 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "subtree", 2024-03-18T16:12:05.288 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "rack", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:12:05.289 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mds_autoscaler", 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.290 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.291 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.292 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.293 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.294 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.295 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.296 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.297 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.298 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirroring", 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.299 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.304 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.305 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.306 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.307 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "nfs", 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.308 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.309 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.310 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.311 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.312 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.313 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.314 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.315 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.316 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "fail_fs": { 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "fail_fs", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.318 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.319 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.320 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.321 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.322 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.323 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.324 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.324 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.325 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.326 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.327 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.328 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator": { 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "orchestrator", 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.329 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: "cephadm", 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: "rook", 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: "test_orchestrator" 2024-03-18T16:12:05.330 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Orchestrator backend", 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.331 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_perf_query", 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.333 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.334 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.335 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.336 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.337 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.338 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.339 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "osd_support", 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.340 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.341 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.342 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.343 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.344 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.345 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.346 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.347 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.348 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "pg_autoscaler", 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.349 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.351 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.352 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.354 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.355 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.356 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:05.357 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.358 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "threshold": { 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "threshold", 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.359 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "3.0", 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1.0", 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "scaling threshold", 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.360 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "progress", 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_pg_recovery_event": { 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:12:05.361 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.362 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.363 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.364 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.365 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.366 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.367 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.368 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.369 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.370 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.371 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "max_completed_events": { 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_completed_events", 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.372 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "50", 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.373 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "sleep_interval": { 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "sleep_interval", 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "5", 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.374 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.375 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "prometheus", 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "cache": { 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "cache", 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.376 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.377 INFO:teuthology.orchestra.run.smithi150.stdout: "exclude_perf_counters": { 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "exclude_perf_counters", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.378 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.379 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.380 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.381 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.382 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.383 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.384 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.385 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.386 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools": { 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools", 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.387 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:12:05.388 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "300", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.389 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "scrape_interval": { 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "scrape_interval", 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.390 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15.0", 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.391 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "::", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.392 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "9283", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.394 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "stale_cache_strategy": { 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stale_cache_strategy", 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "log", 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.395 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_behaviour": { 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_behaviour", 2024-03-18T16:12:05.396 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "default", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "default", 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: "error" 2024-03-18T16:12:05.397 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "standby_error_status_code": { 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "standby_error_status_code", 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.398 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "500", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "400", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "599", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.399 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_support", 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.400 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.401 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.402 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.403 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.404 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.405 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.406 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.407 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_snap_create": { 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:12:05.408 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.409 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.410 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.411 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "trash_purge_schedule": { 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "trash_purge_schedule", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.412 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "restful", 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "enable_auth": { 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enable_auth", 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.415 INFO:teuthology.orchestra.run.smithi150.stdout: "key_file": { 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "key_file", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.416 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.417 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.418 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.419 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.420 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.421 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.422 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.423 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.424 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "server_addr": { 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_addr", 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.425 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "server_port": { 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "server_port", 2024-03-18T16:12:05.426 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.427 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rgw", 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.428 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.431 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.432 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.433 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.434 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rook", 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.437 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.438 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.439 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.440 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.441 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.442 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.443 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.444 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "storage_class": { 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "storage_class", 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "local", 2024-03-18T16:12:05.445 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.446 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "selftest", 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.447 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.448 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.449 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.450 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.451 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.452 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.454 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "roption1": { 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption1", 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.455 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: "roption2": { 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "roption2", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "xyz", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.457 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption1": { 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption1", 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.458 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.459 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption2": { 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption2", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.460 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption3": { 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption3", 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "float", 2024-03-18T16:12:05.461 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.462 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption4": { 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption4", 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.463 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption5": { 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption5", 2024-03-18T16:12:05.464 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.465 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption6": { 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption6", 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.466 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "rwoption7": { 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rwoption7", 2024-03-18T16:12:05.467 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "1", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "42", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.468 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "testkey": { 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testkey", 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.469 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "testlkey": { 2024-03-18T16:12:05.470 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testlkey", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.471 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "testnewline": { 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "testnewline", 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.472 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.473 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snap_schedule", 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "allow_m_granularity": { 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "allow_m_granularity", 2024-03-18T16:12:05.474 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.475 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "dump_on_update": { 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "dump_on_update", 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.476 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.477 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.478 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.479 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.480 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.481 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.482 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.483 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.484 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.485 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "stats", 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.486 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.487 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.488 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.489 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.490 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.492 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.493 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "status", 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.495 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.496 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.497 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.498 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.499 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.500 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.501 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.502 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telegraf", 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "address": { 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "address", 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.503 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.504 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "15", 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.505 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.506 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.507 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.508 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.509 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.510 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.511 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.512 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.513 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "telemetry", 2024-03-18T16:12:05.514 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_basic": { 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_basic", 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_crash": { 2024-03-18T16:12:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_crash", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:12:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_device": { 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_device", 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_ident": { 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_ident", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "channel_perf": { 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "channel_perf", 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "contact": { 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "contact", 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "description": { 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "description", 2024-03-18T16:12:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "device_url": { 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "device_url", 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:12:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "enabled": { 2024-03-18T16:12:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "enabled", 2024-03-18T16:12:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "24", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "8", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "last_opt_revision": { 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "last_opt_revision", 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "1", 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.533 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard": { 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.534 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "leaderboard_description": { 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "leaderboard_description", 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.535 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.537 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.538 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.540 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.541 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.542 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.543 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.544 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "organization": { 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "organization", 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.545 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.546 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "proxy": { 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "proxy", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.547 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "url": { 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "url", 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.548 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.549 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "test_orchestrator", 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.550 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.551 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.552 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.553 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.554 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.555 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.556 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.557 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.558 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "volumes", 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.559 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.560 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.562 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.564 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.565 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "max_concurrent_clones": { 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "max_concurrent_clones", 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.567 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "4", 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.568 INFO:teuthology.orchestra.run.smithi150.stdout: "periodic_async_work": { 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "periodic_async_work", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Periodically check for async work", 2024-03-18T16:12:05.569 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_delay": { 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.570 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "0", 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.571 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "True", 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.572 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix", 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "can_run": true, 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "error_string": "", 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "module_options": { 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "discovery_interval": { 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "discovery_interval", 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "uint", 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.574 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "100", 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "identifier": { 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "identifier", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.576 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "interval": { 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "interval", 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "secs", 2024-03-18T16:12:05.577 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "60", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.578 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "log_level": { 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_level", 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.579 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.580 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster": { 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster", 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.581 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.582 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_cluster_level": { 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_cluster_level", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "info", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [ 2024-03-18T16:12:05.583 INFO:teuthology.orchestra.run.smithi150.stdout: "", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "critical", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "debug", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "error", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "info", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "warning" 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.584 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "log_to_file": { 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "log_to_file", 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "bool", 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 1, 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "False", 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.585 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_host": { 2024-03-18T16:12:05.586 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_host", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.587 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_port": { 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_port", 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "int", 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.588 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "10051", 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "zabbix_sender": { 2024-03-18T16:12:05.589 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "zabbix_sender", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "str", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "level": "advanced", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "flags": 0, 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "min": "", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "max": "", 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "enum_allowed": [], 2024-03-18T16:12:05.590 INFO:teuthology.orchestra.run.smithi150.stdout: "desc": "", 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "long_desc": "", 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "tags": [], 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "see_also": [] 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "services": { 2024-03-18T16:12:05.591 INFO:teuthology.orchestra.run.smithi150.stdout: "dashboard": "https://172.21.15.150:8443/" 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "always_on_modules": { 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "octopus": [ 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "balancer", 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "crash", 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "devicehealth", 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator", 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscaler", 2024-03-18T16:12:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: "progress", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_support", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "status", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "telemetry", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "volumes" 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "pacific": [ 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "balancer", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "crash", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "devicehealth", 2024-03-18T16:12:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscaler", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "progress", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_support", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "status", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "telemetry", 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "volumes" 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "quincy": [ 2024-03-18T16:12:05.594 INFO:teuthology.orchestra.run.smithi150.stdout: "balancer", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "crash", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "devicehealth", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscaler", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "progress", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_support", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "status", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "telemetry", 2024-03-18T16:12:05.595 INFO:teuthology.orchestra.run.smithi150.stdout: "volumes" 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "reef": [ 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "balancer", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "crash", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "devicehealth", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscaler", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "progress", 2024-03-18T16:12:05.596 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_support", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "status", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "telemetry", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "volumes" 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: ], 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "squid": [ 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "balancer", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "crash", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "devicehealth", 2024-03-18T16:12:05.597 INFO:teuthology.orchestra.run.smithi150.stdout: "orchestrator", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "pg_autoscaler", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "progress", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "rbd_support", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "status", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "telemetry", 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "volumes" 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: "last_failure_osd_epoch": 89, 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "active_clients": [ 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "devicehealth", 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3126089862 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.599 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "libcephsqlite", 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2308392019 2024-03-18T16:12:05.600 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "rbd_support", 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.601 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1917150057 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snap_schedule", 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.602 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1509901358 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "snap_schedule", 2024-03-18T16:12:05.603 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 189563406 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.604 INFO:teuthology.orchestra.run.smithi150.stdout: "name": "volumes", 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "addrvec": [ 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v2", 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3359522271 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.605 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:05.606 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-18T16:12:05.606 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T16:12:05.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: from='client.16410 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:05.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: pgmap v374: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-18T16:12:05.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T16:12:05.607 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3880572882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1055192356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:03 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3434856941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: from='client.16410 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: pgmap v374: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3880572882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1055192356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:03 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3434856941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:12:05.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.463+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 msgr2=0x7f5a380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:05.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 0x7f5a380a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009b50 tx=0x7f5a44005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:05.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 msgr2=0x7f5a501985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000e9f0 tx=0x7f5a4000edf0 comp rx=0 tx=0).stop 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 shutdown_connections 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5a380a2970 0x7f5a380a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5a501077e0 0x7f5a5019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a5006a7d0 0x7f5a501985c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 --2- 172.21.15.150:0/3434856941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50069e30 0x7f5a50198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 >> 172.21.15.150:0/3434856941 conn(0x7f5a500764e0 msgr2=0x7f5a500fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:05.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 shutdown_connections 2024-03-18T16:12:05.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:03.464+0000 7f5a54ab2640 1 -- 172.21.15.150:0/3434856941 wait complete. 2024-03-18T16:12:05.616 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T16:12:05.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T16:12:05.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: from='client.16410 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:12:05.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: pgmap v374: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-03-18T16:12:05.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: osdmap e102: 12 total, 12 up, 12 in 2024-03-18T16:12:05.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3880572882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.617 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1055192356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:05.618 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:03 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3434856941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:12:05.868 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:12:05.992 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:05 smithi150 ceph-mon[31946]: pgmap v376: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 236 op/s 2024-03-18T16:12:06.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:05 smithi156 ceph-mon[35919]: pgmap v376: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 236 op/s 2024-03-18T16:12:06.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:05 smithi191 ceph-mon[36784]: pgmap v376: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 236 op/s 2024-03-18T16:12:07.687 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 244, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 232, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 1445712, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 141952, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:07.688 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.039877650, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.011848619, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.030730045, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.003816515, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 232, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.008914764, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000038425 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.689 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1445712, 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 141952 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:07.690 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 2, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 176, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 320, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 186672, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.011472590, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.007599119, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.002007885, 2024-03-18T16:12:07.691 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 173, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.009190918, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000053126 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:07.692 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 186672 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 0, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 0, 2024-03-18T16:12:07.693 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 0, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.000398623, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000338650, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:07.694 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:07.695 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:07.696 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:07.697 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:07.698 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:07.699 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 149034, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 32, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:07.701 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-03-18T16:12:07.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 242, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1408808, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 242, 2024-03-18T16:12:07.703 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 242, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1408808, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:07.704 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:07.705 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T16:12:08.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:07 smithi156 ceph-mon[35919]: pgmap v377: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:08.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:07 smithi150 ceph-mon[31946]: pgmap v377: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:08.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:07 smithi191 ceph-mon[36784]: pgmap v377: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:08.417 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-18T16:12:08.417 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-18T16:12:08.417 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-18T16:12:08.417 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-18T16:12:08.417 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-18T16:12:08.418 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-18T16:12:08.418 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-18T16:12:08.418 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T16:12:08.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T16:12:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:12:09.824 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:09 smithi150 ceph-mon[31946]: pgmap v378: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:09.825 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:09 smithi150 ceph-mon[31946]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T16:12:10.015 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:09 smithi191 ceph-mon[36784]: pgmap v378: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:10.015 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:09 smithi191 ceph-mon[36784]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T16:12:10.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:09 smithi156 ceph-mon[35919]: pgmap v378: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-03-18T16:12:10.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:09 smithi156 ceph-mon[35919]: osdmap e103: 12 total, 12 up, 12 in 2024-03-18T16:12:10.431 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 250, 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 238, 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 1447344, 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 145792, 2024-03-18T16:12:10.432 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.041366870, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.013065744, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.033388579, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.003941893, 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:10.433 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 238, 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.009821316, 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000041266 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:10.434 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1447344, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 145792 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 2, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 180, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 320, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 187376, 2024-03-18T16:12:10.435 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.012068741, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.008182441, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.001927134, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 177, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.011377832, 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000064281 2024-03-18T16:12:10.436 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 187376 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:10.437 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 0, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 0, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 0, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.000338093, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000269139, 2024-03-18T16:12:10.438 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:10.439 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-03-18T16:12:10.440 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:10.441 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:10.442 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:10.443 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:10.444 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 149057, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 32, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:10.445 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:10.446 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-03-18T16:12:10.447 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 248, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1409588, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 248, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 248, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1409588, 2024-03-18T16:12:10.448 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:10.449 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T16:12:11.126 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T16:12:11.126 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T16:12:11.127 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-18T16:12:11.127 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-18T16:12:11.127 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-18T16:12:11.127 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-18T16:12:11.127 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-18T16:12:11.127 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T16:12:11.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T16:12:11.372 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:12:12.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:11 smithi156 ceph-mon[35919]: pgmap v380: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 83 op/s 2024-03-18T16:12:12.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:11 smithi150 ceph-mon[31946]: pgmap v380: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 83 op/s 2024-03-18T16:12:12.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:11 smithi191 ceph-mon[36784]: pgmap v380: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 2.0 MiB/s wr, 83 op/s 2024-03-18T16:12:13.064 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T16:12:13.064 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:13.064 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 0, 2024-03-18T16:12:13.064 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2, 2024-03-18T16:12:13.064 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 0, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 0, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.001271678, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000319867, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:13.065 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:13.066 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 242, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 232, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 1443184, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 142784, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:13.067 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.042292105, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.013409658, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.033093107, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.004089749, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 231, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.009775329, 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000042317 2024-03-18T16:12:13.068 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1443184, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 142784 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.069 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 6, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 180, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 3392, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 187472, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.013654124, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.009381326, 2024-03-18T16:12:13.070 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.002195275, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 178, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.009142533, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000051362 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:13.071 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 187472 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-03-18T16:12:13.072 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:13.073 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:13.074 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:13.075 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:13.076 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:13.076 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:13.077 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:13.077 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:13.077 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:13.077 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 149017, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 31, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:13.078 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-03-18T16:12:13.079 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:13.080 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 244, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1409068, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 244, 2024-03-18T16:12:13.081 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 244, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1409068, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:13.082 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:13.083 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:13.083 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-18T16:12:13.765 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-18T16:12:13.766 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-18T16:12:13.766 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T16:12:13.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T16:12:14.007 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:12:14.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:13 smithi156 ceph-mon[35919]: pgmap v381: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:14.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:13 smithi150 ceph-mon[31946]: pgmap v381: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:14.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:13 smithi191 ceph-mon[36784]: pgmap v381: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:15.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:14 smithi150 ceph-mon[31946]: pgmap v382: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:15.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:14 smithi191 ceph-mon[36784]: pgmap v382: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:15.268 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:14 smithi156 ceph-mon[35919]: pgmap v382: 97 pgs: 97 active+clean; 663 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 1.6 MiB/s wr, 67 op/s 2024-03-18T16:12:15.699 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 4, 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 4, 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 3072, 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 448, 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:15.700 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.001458407, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000034729, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000555768, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 3, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000638181, 2024-03-18T16:12:15.701 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000212727 2024-03-18T16:12:15.702 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.702 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:15.702 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:15.702 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:15.702 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:15.703 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.703 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T16:12:15.703 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T16:12:15.704 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.704 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:15.704 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 248, 2024-03-18T16:12:15.705 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 238, 2024-03-18T16:12:15.705 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 1444816, 2024-03-18T16:12:15.705 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 146624, 2024-03-18T16:12:15.705 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:15.706 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:15.706 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.040464727, 2024-03-18T16:12:15.706 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.013944550, 2024-03-18T16:12:15.706 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.030952438, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.004169736, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 237, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.009438525, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000039825 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:15.707 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1444816, 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 146624 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:15.708 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 2, 2024-03-18T16:12:15.709 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 183, 2024-03-18T16:12:15.709 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 320, 2024-03-18T16:12:15.709 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 187904, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.014556705, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.010126960, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.002083989, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:15.710 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 180, 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.010431579, 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000057953 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:15.711 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 187904 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.712 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2024-03-18T16:12:15.713 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:15.713 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:15.713 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:15.713 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:15.713 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:15.714 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:15.714 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:15.714 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:15.714 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:15.715 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:15.715 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:15.715 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:15.715 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:15.716 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:15.716 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:15.716 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:15.716 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:15.717 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:15.718 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 148994, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 31, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2024-03-18T16:12:15.719 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:15.720 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:15.721 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 250, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1409848, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 250, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2024-03-18T16:12:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 250, 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1409848, 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-18T16:12:15.723 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-18T16:12:16.411 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-18T16:12:16.412 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T16:12:16.412 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T16:12:16.691 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:12:17.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:17 smithi191 ceph-mon[36784]: pgmap v383: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:18.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:17 smithi156 ceph-mon[35919]: pgmap v383: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:18.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:17 smithi150 ceph-mon[31946]: pgmap v383: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:18.855 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 250027, 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 153984, 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 103370009, 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 54642304, 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 10, 2024-03-18T16:12:18.856 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 9.676688519, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 4.079242760, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 7.308820553, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.172239090, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 153982, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2.085750449, 2024-03-18T16:12:18.857 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000013545 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 389355, 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.037861362, 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000097 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:18.858 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 65929, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 91786, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 19521789, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 60064898, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 12, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 7, 2024-03-18T16:12:18.859 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 3.114809588, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 1.574347369, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 1.925286990, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.085132772, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 91785, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.582890520, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000017245 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.860 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 82752, 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.018311852, 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000221 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1523920, 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 152048 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.861 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 4929, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 5139, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1775446, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 15760248, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 10, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 7, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.339254776, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.121736452, 2024-03-18T16:12:18.862 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.079477462, 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.112731050, 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5137, 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.145974705, 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000028416 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.863 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 60, 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000013246, 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000220 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 7040, 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 495824 2024-03-18T16:12:18.864 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 17980, 2024-03-18T16:12:18.865 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 10.065246245, 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000559802 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 8687, 2024-03-18T16:12:18.866 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.465877199, 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000053629 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 52372, 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 27924, 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 27924, 2024-03-18T16:12:18.867 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 33.630141128, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001204345 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 24448, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 11, 2024-03-18T16:12:18.868 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 128, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 5, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 123, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 116, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 41, 2024-03-18T16:12:18.869 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 42, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 4, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 94786, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 36944, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 24445, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 11, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:18.870 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 1695, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 1, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 11, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:18.871 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 234, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:18.872 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 89, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 18832, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:18.873 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 4921, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 3076, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 1427, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 28643 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 126, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 33, 2024-03-18T16:12:18.874 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 22, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 23, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 126, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 36, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 5097, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 5061, 2024-03-18T16:12:18.875 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-18T16:12:18.877 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 28800, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 28800, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 37, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-18T16:12:18.878 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 37, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 4, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 51485354, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-03-18T16:12:18.879 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 8662, 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 15.884720762, 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001833839 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 107, 2024-03-18T16:12:18.880 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 5160, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 5053, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 37, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 39, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 2, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 116, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 5172, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 5056, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 32, 2024-03-18T16:12:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 5160, 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 5128, 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 68332, 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 223612 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 81580, 2024-03-18T16:12:18.882 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 52372, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 1714, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 11885, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5091, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 10.822928023, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002125894 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.883 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000680374, 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000226791 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6408, 2024-03-18T16:12:18.885 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 5.083111025, 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000793244 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.886 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000098902, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000098902 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.887 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.889 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.890 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.891 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-18T16:12:18.892 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 8273, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 4.018397424, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000485724 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1695, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3.892159038, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002296259 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.893 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.894 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.895 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.896 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1356, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.472773598, 2024-03-18T16:12:18.897 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000348653 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-18T16:12:18.898 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.899 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5097, 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 9.339992744, 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001832449 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:18.900 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 2, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2024-03-18T16:12:18.901 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 2, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 3285, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 1642, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 702, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:18.902 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:18.903 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:18.904 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:18.905 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:18.906 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 513535, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 276, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 88, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 1, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:18.907 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 186, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 523990, 2024-03-18T16:12:18.908 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 11364, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-18T16:12:18.909 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 15696, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 48330704, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 33, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 15696, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 15696, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 32.896555770, 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002095855 2024-03-18T16:12:18.910 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 15696, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 16613 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 15696, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 506, 2024-03-18T16:12:18.911 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 15190, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 0, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 10, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 506, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 9168, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 300, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-18T16:12:18.912 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 5084, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 20, 2024-03-18T16:12:18.913 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-18T16:12:18.914 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-18T16:12:18.915 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-18T16:12:18.916 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 103, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 17, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 12, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 12, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 1225, 2024-03-18T16:12:18.917 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 300 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.918 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T16:12:18.919 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 10, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 5061, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 5061, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:18.920 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 320881, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 96761513, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 320881, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 320881, 2024-03-18T16:12:18.921 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 96761513, 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:18.922 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 15696, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 48842193, 2024-03-18T16:12:18.923 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 15696, 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 48842193, 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:18.924 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 15696, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 15696, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 15696, 2024-03-18T16:12:18.925 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 15696, 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:18.926 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 5061, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 511489, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 5061, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:18.927 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 506, 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 511489, 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55635e534500": { 2024-03-18T16:12:18.928 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 65457, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 28800, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 47291050, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 28800, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:18.929 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 8662, 2024-03-18T16:12:18.930 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 47225593, 2024-03-18T16:12:18.930 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:18.930 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:18.930 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:18.930 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:18.931 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.931 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:18.931 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-18T16:12:19.774 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=126 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=33 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=0 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-03-18T16:12:19.775 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T16:12:19.775 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T16:12:19.776 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T16:12:19.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:19 smithi191 ceph-mon[36784]: pgmap v384: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:20.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:19 smithi156 ceph-mon[35919]: pgmap v384: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:20.027 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:12:20.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:19 smithi150 ceph-mon[31946]: pgmap v384: 97 pgs: 97 active+clean; 661 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.6 MiB/s wr, 60 op/s 2024-03-18T16:12:21.846 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:21 smithi156 ceph-mon[35919]: pgmap v385: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-03-18T16:12:21.873 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-18T16:12:21.873 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:21.873 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1066, 2024-03-18T16:12:21.874 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1056, 2024-03-18T16:12:21.874 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3161269, 2024-03-18T16:12:21.874 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 404132, 2024-03-18T16:12:21.874 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:21.874 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.087195845, 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.027359070, 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.045566348, 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.019596556, 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:21.875 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1055, 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.024647928, 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000023362 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.876 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1466400, 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 149728 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:21.877 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 10, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 189, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 5751, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 316938, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-03-18T16:12:21.878 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.009366867, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.005245018, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.002429227, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000089059, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 187, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.017091864, 2024-03-18T16:12:21.879 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000091400 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.880 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 315600 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 367, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 369, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 90886, 2024-03-18T16:12:21.881 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 116713, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.024170591, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.006667652, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.005924231, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.009529739, 2024-03-18T16:12:21.882 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 367, 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.008375232, 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000022820 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.883 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.884 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:21.885 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1956, 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.040850441, 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000020884 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.886 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.887 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.888 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-03-18T16:12:21.889 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 5, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 1, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 4, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 5, 2024-03-18T16:12:21.890 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-03-18T16:12:21.891 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-03-18T16:12:21.892 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 66, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 21133856, 2024-03-18T16:12:21.893 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 6, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:21.894 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:12:21.895 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-03-18T16:12:21.896 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:21.897 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:21.898 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 117, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-18T16:12:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 27, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 8251087, 2024-03-18T16:12:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 8331142, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 8305587, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 117 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-18T16:12:21.901 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 8, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 8, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 0, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 7, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 7, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 0, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 5, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 5, 2024-03-18T16:12:21.902 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 0, 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 48012, 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-03-18T16:12:21.903 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-18T16:12:21.904 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.905 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.906 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.907 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.908 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.909 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.910 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-18T16:12:21.911 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.911 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.911 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.911 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.912 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.913 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:21.914 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.915 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:21.916 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.917 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.918 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.919 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.920 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.921 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.922 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.923 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 2, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 0, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 2086, 2024-03-18T16:12:21.924 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:21.925 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:21.926 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:21.927 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-18T16:12:21.928 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 190487, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 60, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:21.929 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 186, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:21.930 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 62861, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 310, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-18T16:12:21.931 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 1171, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 26, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 4, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 1171, 2024-03-18T16:12:21.932 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1171, 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.445994081, 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000380865 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1171, 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1172 2024-03-18T16:12:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 1171, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 1171, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 728, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-03-18T16:12:21.934 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 440, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-03-18T16:12:21.935 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 2, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-18T16:12:21.936 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-18T16:12:21.937 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-18T16:12:21.938 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 103, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 12, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 5, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 5, 2024-03-18T16:12:21.939 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-18T16:12:21.940 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:12:21.941 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:21.942 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 1439, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 3111690, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 1439, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1439, 2024-03-18T16:12:21.943 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 3111690, 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:21.944 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1171, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1455573, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1171, 2024-03-18T16:12:21.945 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1455573, 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:21.946 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1171, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1171, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1171, 2024-03-18T16:12:21.947 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1171, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:21.948 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-18T16:12:21.949 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x56453510e6e0": { 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:21.950 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-18T16:12:21.951 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:21.952 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-18T16:12:22.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:21 smithi150 ceph-mon[31946]: pgmap v385: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-03-18T16:12:22.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:21 smithi191 ceph-mon[36784]: pgmap v385: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 2.3 MiB/s wr, 83 op/s 2024-03-18T16:12:22.704 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T16:12:22.704 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T16:12:22.705 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-18T16:12:22.705 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-18T16:12:22.705 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-18T16:12:22.705 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-18T16:12:22.705 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T16:12:22.705 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T16:12:22.705 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T16:12:22.955 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:12:24.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:23 smithi156 ceph-mon[35919]: pgmap v386: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:24.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:23 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:23 smithi150 ceph-mon[31946]: pgmap v386: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:24.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:23 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:24.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:23 smithi191 ceph-mon[36784]: pgmap v386: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:24.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:23 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:24.788 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-18T16:12:24.788 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:24.788 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2159, 2024-03-18T16:12:24.788 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2161, 2024-03-18T16:12:24.788 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 25788656, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 686569, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.143260987, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.040094820, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.049709495, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.042855729, 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:24.789 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2159, 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.042790617, 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000019819 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.790 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 917, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 897, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 18448996, 2024-03-18T16:12:24.791 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 352754, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.083663641, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.022431143, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.052204947, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.016870791, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 896, 2024-03-18T16:12:24.792 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.020661956, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000023060 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1511040, 2024-03-18T16:12:24.793 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 149808 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 279, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 456, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 5427601, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 569710, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-03-18T16:12:24.794 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.028338607, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.010971244, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.008992573, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.005589771, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 454, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.013366734, 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000029442 2024-03-18T16:12:24.795 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 482896 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.796 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5363, 2024-03-18T16:12:24.797 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.094871006, 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000017689 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.798 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.799 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.799 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.799 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.799 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-18T16:12:24.799 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-03-18T16:12:24.800 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 5157, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 1, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 5156, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-03-18T16:12:24.801 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 5156, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-03-18T16:12:24.802 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-03-18T16:12:24.803 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:24.804 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:12:24.805 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-03-18T16:12:24.806 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:24.807 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:24.808 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:24.809 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 28801, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-18T16:12:24.810 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 37, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 4, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-03-18T16:12:24.811 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 51492645, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 51485354, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 28801 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.812 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 91, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 5152, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 5061, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 22, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 22, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 0, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 5157, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 5157, 2024-03-18T16:12:24.813 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 0, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 54740, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-18T16:12:24.814 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.815 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.816 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-18T16:12:24.817 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.818 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.819 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.820 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.821 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-18T16:12:24.822 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.822 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.822 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-18T16:12:24.823 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.824 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.825 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.826 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:24.827 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.828 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.829 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.830 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.831 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.832 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.833 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 2, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 0, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 2, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-03-18T16:12:24.834 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 2089, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:24.835 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:24.836 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:24.837 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:24.838 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 217976, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 142, 2024-03-18T16:12:24.839 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:24.840 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 186, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 3932131, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 26657, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-18T16:12:24.841 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 3070, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2024-03-18T16:12:24.842 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 3070, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3070, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.464980062, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000477192 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-18T16:12:24.843 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3070, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3071 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 3070, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 3070, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-18T16:12:24.844 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 1594, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 1474, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-18T16:12:24.845 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:24.846 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-18T16:12:24.847 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-18T16:12:24.848 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 103, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 17, 2024-03-18T16:12:24.849 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 8, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 8, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-03-18T16:12:24.850 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-18T16:12:24.851 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-03-18T16:12:24.852 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 3351, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 49346366, 2024-03-18T16:12:24.853 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 3351, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3351, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 49346366, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.854 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:24.855 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3070, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 47302089, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3070, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 47302089, 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:24.856 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:24.857 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3070, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 3070, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3070, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 3070, 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:24.858 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:24.859 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-18T16:12:24.860 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x560d742ce6e0": { 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-18T16:12:24.861 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-18T16:12:24.862 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:24.863 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-18T16:12:25.389 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: pgmap v387: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:12:25.390 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:12:25.391 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:25 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: pgmap v387: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:12:25.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:12:25.464 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:25 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.592 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: pgmap v387: 97 pgs: 97 active+clean; 662 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 1.4 MiB/s wr, 50 op/s 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:12:25.593 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:25 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-18T16:12:25.632 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T16:12:25.633 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T16:12:25.633 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T16:12:25.881 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:12:27.652 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-18T16:12:27.652 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 253, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 243, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1446800, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 149520, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.036348872, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.009883026, 2024-03-18T16:12:27.653 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.030772664, 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.004257910, 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 242, 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.006890798, 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000028474 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:27.654 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1446800, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 149520 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2, 2024-03-18T16:12:27.655 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 186, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 320, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 188432, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.011084058, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.005436577, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.003881716, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:27.656 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 183, 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.014244569, 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000077839 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:27.657 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 188432 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 4, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 4, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3072, 2024-03-18T16:12:27.658 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 448, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.000537721, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000034510, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000487239, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2024-03-18T16:12:27.659 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000720761, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000240253 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T16:12:27.660 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 0, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:27.661 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:27.662 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:27.663 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:27.664 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 148923, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 29, 2024-03-18T16:12:27.665 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:27.666 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 2, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:27.667 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 255, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 1411135, 2024-03-18T16:12:27.668 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 255, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 255, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1411135, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-18T16:12:27.669 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-18T16:12:27.670 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-18T16:12:27.670 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:27.670 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-18T16:12:27.670 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-18T16:12:27.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:27 smithi191 ceph-mon[36784]: pgmap v388: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-03-18T16:12:28.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:27 smithi156 ceph-mon[35919]: pgmap v388: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-03-18T16:12:28.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:27 smithi150 ceph-mon[31946]: pgmap v388: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 2.0 MiB/s wr, 75 op/s 2024-03-18T16:12:28.456 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-18T16:12:28.457 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-18T16:12:28.457 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T16:12:28.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T16:12:28.869 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:12:29.775 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:29 smithi191 ceph-mon[36784]: pgmap v389: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.4 MiB/s wr, 52 op/s 2024-03-18T16:12:30.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:29 smithi156 ceph-mon[35919]: pgmap v389: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.4 MiB/s wr, 52 op/s 2024-03-18T16:12:30.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:29 smithi150 ceph-mon[31946]: pgmap v389: 97 pgs: 97 active+clean; 597 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 1.4 MiB/s wr, 52 op/s 2024-03-18T16:12:31.551 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T16:12:31.551 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:31.551 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 415542, 2024-03-18T16:12:31.551 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 572101, 2024-03-18T16:12:31.551 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 116920123, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 903226247, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 15, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 19.631618638, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 9.809446887, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 12.676438815, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 1.341990370, 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:31.552 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 572099, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 11.231562040, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000019632 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 488537, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.089015016, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000182 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.553 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 39466, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 39648, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 9868970, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 367068737, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 10, 2024-03-18T16:12:31.554 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 4.321190507, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 2.132740259, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.719135757, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 1.199867752, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 39646, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.880556540, 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000047433 2024-03-18T16:12:31.555 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 17, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000002705, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000159 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 5568, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 491296 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.556 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 288242, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 501653, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 57575511, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 359738439, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 7, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 18.892495039, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 8.467381918, 2024-03-18T16:12:31.557 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 10.792680297, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.831004018, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 501652, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 9.553313046, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000019043 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 741250, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.109187772, 2024-03-18T16:12:31.558 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000147 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 1521584, 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 156576 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.559 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 131771, 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 133.169711825, 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001010614 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:31.560 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 53479, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 10.051116427, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000187945 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 279881, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 277561, 2024-03-18T16:12:31.561 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 277561, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 457.833835127, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001649489 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 2320, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:31.562 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 190, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 13, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 17, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 2690, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 1145, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 1517, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 28, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 1055, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 14, 2024-03-18T16:12:31.563 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 126, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 126, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 6, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 436871, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 260960, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 2320, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 12, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:31.564 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 4618, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 1, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 11, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 1, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 152, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:31.565 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 4, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 234, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:31.566 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 5, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 13082, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 118055, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:31.567 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 66544, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 30155, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 15767, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 193965 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 3, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 33, 2024-03-18T16:12:31.568 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 22, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 24, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 146, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 64, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 37289, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 37225, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:31.569 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:31.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 1, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 30, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 180026, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 110611, 2024-03-18T16:12:31.571 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 110611, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 69415, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 12290, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 285, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 157, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 157, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 128, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 6, 2024-03-18T16:12:31.572 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 17, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 435449099, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 945278292, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 0, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 67073, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 135.535957985, 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002020723 2024-03-18T16:12:31.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 0 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 800, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 37910, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 37110, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 61, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 152, 2024-03-18T16:12:31.574 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 91, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 2587, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 77651, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 75064, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 126, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 40586, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 40460, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 77916, 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 223612 2024-03-18T16:12:31.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 422098, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 11886, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 279881, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 4327, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 23788, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 37290, 2024-03-18T16:12:31.576 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 64.334587332, 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001725250 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 15007, 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 61.284824613, 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.004083749 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.577 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T16:12:31.578 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 82907, 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 45.707934831, 2024-03-18T16:12:31.579 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000551315 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.580 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1, 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000133773, 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000133773 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:31.582 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.583 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.583 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T16:12:31.583 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.584 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.585 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T16:12:31.586 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 97885, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 200.840422498, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002051799 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 7169, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 16.570687342, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002311436 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:31.587 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.588 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.589 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 5, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.047222871, 2024-03-18T16:12:31.590 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.009444574 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 8, 2024-03-18T16:12:31.591 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.004300505, 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000537563 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T16:12:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T16:12:31.593 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 37289, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 69.043721362, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001851584 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.594 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 2, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 1, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 1, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T16:12:31.595 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 37994, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 37994, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 751, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:31.596 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:31.597 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:31.598 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:31.599 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 1154331, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 880, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:31.600 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 186, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:31.601 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 4505078, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 78320, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T16:12:31.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 108043, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 950478159, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 33, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 108043, 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T16:12:31.603 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 108043, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 245.942626645, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002276340 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 108043, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 109536 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.604 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 108043, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 1010, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 107033, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 46, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 34, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 1010, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 68083, 2024-03-18T16:12:31.605 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 425, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 13, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 37640, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 0, 2024-03-18T16:12:31.606 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 68, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:31.607 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T16:12:31.608 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T16:12:31.609 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 103, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 17, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 12, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 1659, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 0, 2024-03-18T16:12:31.610 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 558 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 1, 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T16:12:31.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 10, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 37225, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 37225, 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:31.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 743246, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 123382984, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 743246, 2024-03-18T16:12:31.613 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 743246, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 123382984, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 108043, 2024-03-18T16:12:31.615 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 954242036, 2024-03-18T16:12:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 108043, 2024-03-18T16:12:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 954242036, 2024-03-18T16:12:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.618 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.618 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.618 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:31.618 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 108043, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 108043, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 108043, 2024-03-18T16:12:31.619 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 108043, 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:31.620 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 37225, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 3763877, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 37225, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1010, 2024-03-18T16:12:31.621 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 3763877, 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x55e929644500": { 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 74276, 2024-03-18T16:12:31.622 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 180026, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 941083988, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 180026, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 67073, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 941009712, 2024-03-18T16:12:31.623 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:31.624 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T16:12:31.940 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:31 smithi191 ceph-mon[36784]: pgmap v390: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-03-18T16:12:32.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:31 smithi156 ceph-mon[35919]: pgmap v390: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-03-18T16:12:32.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:31 smithi150 ceph-mon[31946]: pgmap v390: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 2.0 MiB/s wr, 79 op/s 2024-03-18T16:12:32.538 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-03-18T16:12:32.538 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-03-18T16:12:32.538 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=3 2024-03-18T16:12:32.539 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=33 2024-03-18T16:12:32.539 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-18T16:12:32.539 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-18T16:12:32.539 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T16:12:32.539 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T16:12:32.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T16:12:32.886 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:12:34.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:33 smithi156 ceph-mon[35919]: pgmap v391: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:34.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:33 smithi150 ceph-mon[31946]: pgmap v391: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:34.177 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:33 smithi191 ceph-mon[36784]: pgmap v391: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:35.042 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T16:12:35.042 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:35.042 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 4, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 4, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 3072, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 448, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 4, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 0, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.001984494, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.000040196, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.000814149, 2024-03-18T16:12:35.043 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000893119, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000297706 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:35.044 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 105, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 104, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 169920, 2024-03-18T16:12:35.045 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 63248, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 2, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 1, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.012027737, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.003692424, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.006592115, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.001906463, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 103, 2024-03-18T16:12:35.046 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.003681288, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000035740 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 169920, 2024-03-18T16:12:35.047 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 63248 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 79, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 160, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 91664, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 3, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 2, 2024-03-18T16:12:35.048 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.004371345, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.002552368, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.001206276, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 77, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.001975539, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000025656 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.049 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 91664 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T16:12:35.050 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:35.051 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:35.052 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:35.053 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:35.054 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 148823, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 27, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:35.055 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 928, 2024-03-18T16:12:35.056 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 2, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:35.057 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 106, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 155121, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 106, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 106, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 155121, 2024-03-18T16:12:35.058 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:35.059 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T16:12:35.735 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-03-18T16:12:35.735 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-03-18T16:12:35.736 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.f 2024-03-18T16:12:35.736 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.f 2024-03-18T16:12:35.736 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-03-18T16:12:35.736 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-03-18T16:12:35.736 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-03-18T16:12:35.736 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T16:12:35.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T16:12:35.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:35 smithi191 ceph-mon[36784]: pgmap v392: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:36.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:35 smithi156 ceph-mon[35919]: pgmap v392: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:36.027 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:12:36.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:35 smithi150 ceph-mon[31946]: pgmap v392: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 1.2 MiB/s wr, 52 op/s 2024-03-18T16:12:37.774 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:37 smithi191 ceph-mon[36784]: pgmap v393: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.7 MiB/s wr, 69 op/s 2024-03-18T16:12:37.778 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T16:12:37.778 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:37.778 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 3117, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 2702, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 1147365, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 5424824, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 19, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.231516612, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.072787745, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.087864684, 2024-03-18T16:12:37.779 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.062698618, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2695, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.601799664, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000223302 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 3648, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000582329, 2024-03-18T16:12:37.780 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000159 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 369360, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 112448 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 236982, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 119202, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 100795320, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 21391885, 2024-03-18T16:12:37.781 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 111, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 6, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 10.427253838, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 4.627326130, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 8.014571187, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.024560824, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 119203, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 2.300602747, 2024-03-18T16:12:37.782 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000019299 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 355248, 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.042833847, 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000120 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 88592, 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 135792 2024-03-18T16:12:37.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 40804, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 53844, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 12640689, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 14075657, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 15, 2024-03-18T16:12:37.784 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 9, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 1.918713924, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.933794388, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 1.208504455, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.051030239, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 53837, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 1.537719846, 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000028562 2024-03-18T16:12:37.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 52846, 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.013208405, 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000249 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 1588528, 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 551888 2024-03-18T16:12:37.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1865, 2024-03-18T16:12:37.787 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.803117266, 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000430625 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 262, 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.014727014, 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000056209 2024-03-18T16:12:37.788 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 26022, 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 1345, 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1345, 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 14.657266251, 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.010897595 2024-03-18T16:12:37.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 24677, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 277, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T16:12:37.790 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 29, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 4, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 15, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 10, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 22, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 183, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 8, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 8, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 4, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 26936, 2024-03-18T16:12:37.791 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 1416, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 24676, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 17, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 19, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T16:12:37.792 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 1, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 152, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 66, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 21133856, 2024-03-18T16:12:37.793 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 6, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:37.794 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 12798, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 13359, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 13034, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 154, 2024-03-18T16:12:37.795 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 68, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 798 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 22, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 260, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 131, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 272, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 47, 2024-03-18T16:12:37.796 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 159, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 159, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:37.797 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 1, 2024-03-18T16:12:37.798 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 1071, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 1057, 2024-03-18T16:12:37.799 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 1057, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 14, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 2, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 56, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 55, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 55, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 1, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 1, 2024-03-18T16:12:37.800 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 1, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 8251087, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 8382252, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 0, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 543, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.974959337, 2024-03-18T16:12:37.801 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001795505 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 0 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 33, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 287, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 254, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 44, 2024-03-18T16:12:37.802 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 201, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 157, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 29, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 590, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 561, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 8, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 263, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 255, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 66688, 2024-03-18T16:12:37.803 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 231804 2024-03-18T16:12:37.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.804 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T16:12:37.804 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 26539, 2024-03-18T16:12:37.804 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 16, 2024-03-18T16:12:37.804 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 26022, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 65, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 17, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 230, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.375346316, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001631940 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 122, 2024-03-18T16:12:37.805 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.022747868, 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000186457 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T16:12:37.806 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 421, 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 12.456982248, 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.029589031 2024-03-18T16:12:37.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 12, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.002229377, 2024-03-18T16:12:37.810 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000185781 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 20, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.026992280, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001349614 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.811 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 12, 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.496495152, 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.041374596 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T16:12:37.812 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000214303, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000214303 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 307, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.758634826, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002471123 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.813 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 42, 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.089479688, 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002130468 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.815 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 6, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.107584300, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.017930716 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.816 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000804119, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000804119 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.817 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 4, 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.001151528, 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000287882 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T16:12:37.818 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 8, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.017977164, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.002247145 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T16:12:37.819 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 159, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.300627082, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001890736 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.820 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 2, 2024-03-18T16:12:37.821 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 6, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 4, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 2, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 28, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 14, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 729, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.822 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 360, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 360, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:37.823 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:37.824 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:37.825 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 315273, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 142, 2024-03-18T16:12:37.826 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 352, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 4, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 186, 2024-03-18T16:12:37.827 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 195794, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 1314, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:37.828 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 1883, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 5273685, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 37, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 1883, 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T16:12:37.829 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1883, 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 21.260612264, 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.011290819 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1883, 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 2627 2024-03-18T16:12:37.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 1883, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 6, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 1877, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 258, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 98, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 6, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 549, 2024-03-18T16:12:37.831 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 194, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 671, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 0, 2024-03-18T16:12:37.832 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 196, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:37.833 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T16:12:37.834 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 103, 2024-03-18T16:12:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 18, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 12, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 12, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 643, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 0, 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 12 2024-03-18T16:12:37.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 1, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.837 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 10, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 10, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 159, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 159, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.838 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 280899, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 89867077, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 280899, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:37.839 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 280899, 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 89867077, 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.840 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:37.841 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1883, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 5289744, 2024-03-18T16:12:37.842 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1883, 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 5289744, 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:37.843 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 1883, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 1883, 2024-03-18T16:12:37.844 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 1883, 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 1883, 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:37.845 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 159, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 16059, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 159, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:37.846 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 6, 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 16059, 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x55e0516c8500": { 2024-03-18T16:12:37.847 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 1071, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 4187948, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 1071, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 543, 2024-03-18T16:12:37.848 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 4187948, 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:37.849 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T16:12:38.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:37 smithi156 ceph-mon[35919]: pgmap v393: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.7 MiB/s wr, 69 op/s 2024-03-18T16:12:38.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:37 smithi150 ceph-mon[31946]: pgmap v393: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 1.7 MiB/s wr, 69 op/s 2024-03-18T16:12:38.606 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-03-18T16:12:38.606 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-18T16:12:38.606 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=22 2024-03-18T16:12:38.606 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=260 2024-03-18T16:12:38.606 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-18T16:12:38.607 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=12 2024-03-18T16:12:38.607 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=6 2024-03-18T16:12:38.607 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T16:12:38.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T16:12:38.862 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:12:40.022 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:39 smithi156 ceph-mon[35919]: pgmap v394: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-03-18T16:12:40.034 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:39 smithi191 ceph-mon[36784]: pgmap v394: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-03-18T16:12:40.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:39 smithi150 ceph-mon[31946]: pgmap v394: 97 pgs: 97 active+clean; 596 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-03-18T16:12:40.674 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 1514, 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1516, 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 386286954, 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 481516, 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 8, 2024-03-18T16:12:40.675 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.294430460, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.026099661, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.228554434, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.030377452, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1514, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.031827737, 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000021022 2024-03-18T16:12:40.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:12:40.677 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 906, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 1091, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 229153209, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 776973, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 9, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 7, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.171115123, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.023285486, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.121965602, 2024-03-18T16:12:40.678 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.019324639, 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 1089, 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.032060583, 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000029440 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.679 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 490768 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_messages": 2158, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages": 2138, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_bytes": 329219651, 2024-03-18T16:12:40.680 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_bytes": 750246, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_created_connections": 5, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_active_connections": 4, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_total_time": 0.319920601, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_send_time": 0.045983883, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_recv_time": 0.228819035, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_running_fast_dispatch_time": 0.044932008, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 2137, 2024-03-18T16:12:40.681 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.045841054, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000021451 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_recv_encrypted_bytes": 1514384, 2024-03-18T16:12:40.682 INFO:teuthology.orchestra.run.smithi191.stdout: "msgr_send_encrypted_bytes": 155904 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "cct": { 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "total_workers": 1, 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "unhealthy_workers": 0 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-MDSRank": { 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:40.683 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 6560, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.185528601, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000028281 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "finisher-PurgeQueue": { 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "queue_len": 0, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "complete_latency": { 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.684 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "mds": { 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "request": 0, 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "reply": 0, 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "reply_latency": { 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_reply": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "forward": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_complete": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_fetch_keys": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_commit": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_split": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_merge": 0, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes": 37985, 2024-03-18T16:12:40.686 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_top": 15941, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_bottom": 22044, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pin_tail": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_pinned": 15213, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_expired": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "inodes_with_caps": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "caps": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "subtrees": 4, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_hit": 0, 2024-03-18T16:12:40.687 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_forward": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_discover": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_remote_ino": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "traverse_lock": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "load_cent": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "q": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "exported": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "exported_inodes": 0, 2024-03-18T16:12:40.688 INFO:teuthology.orchestra.run.smithi191.stdout: "imported": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "imported_inodes": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_dir_fetch": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "openino_peer_discover": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rfiles": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rbytes": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "root_rsnaps": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_set_tag": 0, 2024-03-18T16:12:40.689 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "scrub_file_inodes": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:12:40.690 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_cap_release": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "process_request_cap_release": 0 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_cache": { 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_update_receipt": 0, 2024-03-18T16:12:40.691 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_try_discover": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_send_discover": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "dir_handle_discover": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_delayed": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_created": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_enqueued": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_reintegrated": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "strays_migrated": 0, 2024-03-18T16:12:40.692 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_processing": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_started": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "recovery_completed": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_exportdir": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_flush": 0, 2024-03-18T16:12:40.693 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_fragstats": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "ireq_inodestats": 0 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_log": { 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "evlrg": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "evadd": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "evex": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "evtrm": 0, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "ev": 180181, 2024-03-18T16:12:40.694 INFO:teuthology.orchestra.run.smithi191.stdout: "evexg": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "evexd": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segadd": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segex": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segtrm": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "seg": 128, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segmjr": 16, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segexg": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "segexd": 0, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "expos": 435449099, 2024-03-18T16:12:40.695 INFO:teuthology.orchestra.run.smithi191.stdout: "wrpos": 946142018, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "rdpos": 946135200, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "jlat": { 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "replayed": 180181 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_mem": { 2024-03-18T16:12:40.696 INFO:teuthology.orchestra.run.smithi191.stdout: "ino": 629, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "ino+": 37927, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "ino-": 37298, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dir": 61, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dir+": 151, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dir-": 90, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dn": 37985, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dn+": 37996, 2024-03-18T16:12:40.697 INFO:teuthology.orchestra.run.smithi191.stdout: "dn-": 11, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "cap": 0, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "cap+": 0, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "cap-": 0, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "rss": 89592, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "heap": 215420 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_server": { 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_client_request": 0, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "dispatch_server_request": 0, 2024-03-18T16:12:40.698 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_request": 0, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_client_session": 0, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "handle_peer_request": 0, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "req_create_latency": { 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getattr_latency": { 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.699 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getfilelock_latency": { 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "req_link_latency": { 2024-03-18T16:12:40.700 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookup_latency": { 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookuphash_latency": { 2024-03-18T16:12:40.701 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupino_latency": { 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.702 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupname_latency": { 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupparent_latency": { 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lookupsnap_latency": { 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "req_lssnap_latency": { 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.705 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mkdir_latency": { 2024-03-18T16:12:40.705 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.705 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mknod_latency": { 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "req_mksnap_latency": { 2024-03-18T16:12:40.706 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "req_open_latency": { 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.707 INFO:teuthology.orchestra.run.smithi191.stdout: "req_readdir_latency": { 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rename_latency": { 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "req_renamesnap_latency": { 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "req_snapdiff_latency": { 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.709 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmdir_latency": { 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmsnap_latency": { 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.710 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "req_rmxattr_latency": { 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setattr_latency": { 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.711 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setdirlayout_latency": { 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setfilelock_latency": { 2024-03-18T16:12:40.712 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setlayout_latency": { 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.713 INFO:teuthology.orchestra.run.smithi191.stdout: "req_setxattr_latency": { 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "req_symlink_latency": { 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.714 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "req_unlink_latency": { 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:12:40.715 INFO:teuthology.orchestra.run.smithi191.stdout: "req_getvxattr_latency": { 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_sessions": { 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "session_count": 1, 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "session_add": 2, 2024-03-18T16:12:40.716 INFO:teuthology.orchestra.run.smithi191.stdout: "session_remove": 1, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_open": 2, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "sessions_stale": 0, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "total_load": 0, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "average_load": 0, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "avg_session_uptime": 3495, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "mdthresh_evicted": 0 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "mempool": { 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:12:40.717 INFO:teuthology.orchestra.run.smithi191.stdout: "bloom_filter_items": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:12:40.718 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_extent_items": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_blob_items": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:12:40.719 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_txc_items": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluestore_writing_items": 0, 2024-03-18T16:12:40.720 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_bytes": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_items": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_bytes": 268345, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_anon_items": 860, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "buffer_meta_items": 0, 2024-03-18T16:12:40.721 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_bytes": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_items": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_mapbl_items": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_pglog_items": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_bytes": 8160, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_items": 186, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:12:40.722 INFO:teuthology.orchestra.run.smithi191.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_bytes": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "pgmap_items": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_bytes": 28509930, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "mds_co_items": 169625, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_bytes": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_1_items": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_bytes": 0, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "unittest_2_items": 0 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.723 INFO:teuthology.orchestra.run.smithi191.stdout: "objecter": { 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_active": 0, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_laggy": 0, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send": 4283, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_send_bytes": 21, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_resend": 3, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_reply": 4283, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "op_latency": { 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 4283, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 4.635917525, 2024-03-18T16:12:40.724 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.001082399 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "op_inflight": 0, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "oplen_avg": { 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 4283, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 4284 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "op": 4283, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "op_r": 4283, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "op_w": 0, 2024-03-18T16:12:40.725 INFO:teuthology.orchestra.run.smithi191.stdout: "op_rmw": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "op_pg": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_stat": 2584, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_create": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_read": 1697, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_write": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writefull": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_writesame": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_append": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_zero": 0, 2024-03-18T16:12:40.726 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_truncate": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_delete": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_mapext": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_sparse_read": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_clonerange": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_getxattr": 1, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_setxattr": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_rmxattr": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:12:40.727 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_call": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_watch": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_notify": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "osdop_other": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_active": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_send": 0, 2024-03-18T16:12:40.728 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_resend": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "linger_ping": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_active": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_send": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolop_resend": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_active": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_send": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "poolstat_resend": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_active": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_send": 0, 2024-03-18T16:12:40.729 INFO:teuthology.orchestra.run.smithi191.stdout: "statfs_resend": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "command_active": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "command_send": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "command_resend": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "map_epoch": 103, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "map_full": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "map_inc": 17, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_sessions": 11, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_open": 11, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_session_close": 0, 2024-03-18T16:12:40.730 INFO:teuthology.orchestra.run.smithi191.stdout: "osd_laggy": 0, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_wr": 0, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_rd": 2, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_del": 0 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "oft": { 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_objs": 0, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_updates": 0, 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: "omap_total_removes": 0 2024-03-18T16:12:40.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "osd-slow-ops": { 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "slow_ops_count": 0 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "purge_queue": { 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops": 0, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing": 0, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executing_high_water": 0, 2024-03-18T16:12:40.732 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed_ops": 0, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_executed": 0, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "pq_item_in_journal": 0 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 4574, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 944230350, 2024-03-18T16:12:40.733 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 4574, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 4574, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 944230350, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.734 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_bytes": { 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 104857600, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:40.735 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 4283, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 941973142, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 4283, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 941973142, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.736 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-objecter_ops": { 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 1024, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:40.737 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 4283, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 4283, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 4283, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 4283, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.738 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:40.739 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.740 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "throttle-write_buf_throttle-0x56112d55c6e0": { 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "val": 0, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "max": 3758096384, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "get_started": 0, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "get": 0, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "get_sum": 0, 2024-03-18T16:12:40.741 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_fail": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "get_or_fail_success": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "take": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "take_sum": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "put": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "put_sum": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "wait": { 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "avgcount": 0, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "sum": 0.000000000, 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: "avgtime": 0.000000000 2024-03-18T16:12:40.742 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.743 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-03-18T16:12:40.743 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-18T16:12:41.471 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-18T16:12:41.472 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-18T16:12:41.472 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T16:12:41.482 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T16:12:41.482 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T16:12:41.483 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:12:41.503 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-18T16:12:41.503 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:12:41.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:12:41.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 -- 172.21.15.150:0/1294599484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 msgr2=0x7f2930110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:41.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 --2- 172.21.15.150:0/1294599484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f2930110450 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2928009ca0 tx=0x7f292802c180 comp rx=0 tx=0).stop 2024-03-18T16:12:41.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 -- 172.21.15.150:0/1294599484 shutdown_connections 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 --2- 172.21.15.150:0/1294599484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f2930110450 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 --2- 172.21.15.150:0/1294599484 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29300744b0 0x7f293010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 --2- 172.21.15.150:0/1294599484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2930073b90 0x7f2930073f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 -- 172.21.15.150:0/1294599484 >> 172.21.15.150:0/1294599484 conn(0x7f29300fd250 msgr2=0x7f29300ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 -- 172.21.15.150:0/1294599484 shutdown_connections 2024-03-18T16:12:41.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.714+0000 7f2936044640 1 -- 172.21.15.150:0/1294599484 wait complete. 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 Processor -- start 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 -- start start 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 0x7f293019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 0x7f293019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2930113d80 con 0x7f293010e060 2024-03-18T16:12:41.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2930113c00 con 0x7f29300744b0 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2936044640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2930113f00 con 0x7f2930073b90 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45930/0 (socket says 172.21.15.150:45930) 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 -- 172.21.15.150:0/1524999265 learned_addr learned my addr 172.21.15.150:0/1524999265 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292f7fe640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 0x7f293019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2927fff640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 0x7f293019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 -- 172.21.15.150:0/1524999265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 msgr2=0x7f293019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:41.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 0x7f293019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 -- 172.21.15.150:0/1524999265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 msgr2=0x7f293019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:41.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 0x7f293019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292ffff640 1 -- 172.21.15.150:0/1524999265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293019f2f0 con 0x7f293010e060 2024-03-18T16:12:41.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f292f7fe640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 0x7f293019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.715+0000 7f2927fff640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 0x7f293019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f292ffff640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f292802c790 tx=0x7f2928035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928039040 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2928044070 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293019f580 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928032940 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.716+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29301a6e70 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.717+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2930068a20 con 0x7f293010e060 2024-03-18T16:12:41.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.718+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f2928032ae0 con 0x7f293010e060 2024-03-18T16:12:41.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.718+0000 7f292d7fa640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 0x7f291c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:41.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.718+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f29280f12c0 con 0x7f293010e060 2024-03-18T16:12:41.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.718+0000 7f292f7fe640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 0x7f291c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:41.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.719+0000 7f292f7fe640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 0x7f291c0a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2920009c20 tx=0x7f2920005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:41.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.723+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29280be1c0 con 0x7f293010e060 2024-03-18T16:12:41.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.901+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29301091e0 con 0x7f293010e060 2024-03-18T16:12:41.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.902+0000 7f292d7fa640 1 -- 172.21.15.150:0/1524999265 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23202 (secure 0 0 0) 0x7f29280bdac0 con 0x7f293010e060 2024-03-18T16:12:41.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:41.905 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":103,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T16:12:09.425191+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2222781423":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6800/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/1896654966":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/3130851108":"2024-03-18T17:04:57.464453+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3143158665":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3506843134":"2024-03-19T16:00:04.468782+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.904+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 msgr2=0x7f291c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.904+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 0x7f291c0a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2920009c20 tx=0x7f2920005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.904+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 msgr2=0x7f293019ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.904+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f292802c790 tx=0x7f2928035e00 comp rx=0 tx=0).stop 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 shutdown_connections 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f291c0a2970 0x7f291c0a4e30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293010e060 0x7f293019ebc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29300744b0 0x7f293019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 --2- 172.21.15.150:0/1524999265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2930073b90 0x7f293019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:41.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 >> 172.21.15.150:0/1524999265 conn(0x7f29300fd250 msgr2=0x7f29300fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:41.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 shutdown_connections 2024-03-18T16:12:41.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:41.905+0000 7f2936044640 1 -- 172.21.15.150:0/1524999265 wait complete. 2024-03-18T16:12:41.916 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:41 smithi150 ceph-mon[31946]: pgmap v395: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.1 MiB/s wr, 48 op/s 2024-03-18T16:12:41.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T15:53:17.702512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T15:57:54.584892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T15:57:55.700788+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:12:41.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:12:42.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:41 smithi156 ceph-mon[35919]: pgmap v395: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.1 MiB/s wr, 48 op/s 2024-03-18T16:12:42.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 -- 172.21.15.150:0/2390208635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580102110 msgr2=0x7fe58010e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 --2- 172.21.15.150:0/2390208635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580102110 0x7fe58010e620 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe578009c10 tx=0x7fe57802bef0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 -- 172.21.15.150:0/2390208635 shutdown_connections 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 --2- 172.21.15.150:0/2390208635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580102110 0x7fe58010e620 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 --2- 172.21.15.150:0/2390208635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 0x7fe580101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 --2- 172.21.15.150:0/2390208635 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580107760 0x7fe580107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 -- 172.21.15.150:0/2390208635 >> 172.21.15.150:0/2390208635 conn(0x7fe5800fd240 msgr2=0x7fe5800ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 -- 172.21.15.150:0/2390208635 shutdown_connections 2024-03-18T16:12:42.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.022+0000 7fe586b16640 1 -- 172.21.15.150:0/2390208635 wait complete. 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 Processor -- start 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 -- start start 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 0x7fe580198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580102110 0x7fe580198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe580111b30 con 0x7fe580107760 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5801119b0 con 0x7fe580101750 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe586b16640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fe580111cb0 con 0x7fe580102110 2024-03-18T16:12:42.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe57ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580102110 0x7fe580198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45950/0 (socket says 172.21.15.150:45950) 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 -- 172.21.15.150:0/2104511545 learned_addr learned my addr 172.21.15.150:0/2104511545 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5848b2640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 0x7fe580198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 -- 172.21.15.150:0/2104511545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580102110 msgr2=0x7fe580198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580102110 0x7fe580198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 -- 172.21.15.150:0/2104511545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 msgr2=0x7fe580198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 0x7fe580198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 -- 172.21.15.150:0/2104511545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58019d040 con 0x7fe580107760 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe5850b3640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe578004af0 tx=0x7fe5780336b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.023+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578043020 con 0x7fe580107760 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.024+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57802ca80 con 0x7fe580107760 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.024+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58019d2d0 con 0x7fe580107760 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.024+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57803e450 con 0x7fe580107760 2024-03-18T16:12:42.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.024+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5801a4bc0 con 0x7fe580107760 2024-03-18T16:12:42.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.025+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe580068a20 con 0x7fe580107760 2024-03-18T16:12:42.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.026+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fe578046020 con 0x7fe580107760 2024-03-18T16:12:42.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.026+0000 7fe57dffb640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 0x7fe5700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.026+0000 7fe5848b2640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 0x7fe5700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.026+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7fe5780f19f0 con 0x7fe580107760 2024-03-18T16:12:42.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.027+0000 7fe5848b2640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 0x7fe5700a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe574005fd0 tx=0x7fe574009260 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.030+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe57804d020 con 0x7fe580107760 2024-03-18T16:12:42.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.204+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe580109ce0 con 0x7fe580107760 2024-03-18T16:12:42.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.205+0000 7fe57dffb640 1 -- 172.21.15.150:0/2104511545 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7fe5780be860 con 0x7fe580107760 2024-03-18T16:12:42.207 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 msgr2=0x7fe5700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 0x7fe5700a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe574005fd0 tx=0x7fe574009260 comp rx=0 tx=0).stop 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 msgr2=0x7fe58019c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe578004af0 tx=0x7fe5780336b0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 shutdown_connections 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fe5700a2970 0x7fe5700a4e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe580107760 0x7fe58019c910 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe580102110 0x7fe580198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 --2- 172.21.15.150:0/2104511545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe580101750 0x7fe580198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 >> 172.21.15.150:0/2104511545 conn(0x7fe5800fd240 msgr2=0x7fe5800fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 shutdown_connections 2024-03-18T16:12:42.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.207+0000 7fe586b16640 1 -- 172.21.15.150:0/2104511545 wait complete. 2024-03-18T16:12:42.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:41 smithi191 ceph-mon[36784]: pgmap v395: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 1.1 MiB/s wr, 48 op/s 2024-03-18T16:12:42.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:12:42.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 -- 172.21.15.150:0/3222041862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064101050 msgr2=0x7f206410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 --2- 172.21.15.150:0/3222041862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064101050 0x7f206410d560 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f205c009cd0 tx=0x7f205c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 -- 172.21.15.150:0/3222041862 shutdown_connections 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 --2- 172.21.15.150:0/3222041862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064101050 0x7f206410d560 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 --2- 172.21.15.150:0/3222041862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064100690 0x7f2064100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 --2- 172.21.15.150:0/3222041862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 0x7f2064106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 -- 172.21.15.150:0/3222041862 >> 172.21.15.150:0/3222041862 conn(0x7f2064076a60 msgr2=0x7f2064076e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 -- 172.21.15.150:0/3222041862 shutdown_connections 2024-03-18T16:12:42.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.323+0000 7f206a8f7640 1 -- 172.21.15.150:0/3222041862 wait complete. 2024-03-18T16:12:42.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 Processor -- start 2024-03-18T16:12:42.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 -- start start 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064101050 0x7f20641985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 0x7f206419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2064111bb0 con 0x7f2064100690 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2064111a30 con 0x7f2064101050 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f206a8f7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f2064111d30 con 0x7f20641066a0 2024-03-18T16:12:42.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f2063fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f2063fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45962/0 (socket says 172.21.15.150:45962) 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f2063fff640 1 -- 172.21.15.150:0/4285472886 learned_addr learned my addr 172.21.15.150:0/4285472886 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f20637fe640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064101050 0x7f20641985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.324+0000 7f2068e94640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 0x7f206419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 -- 172.21.15.150:0/4285472886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 msgr2=0x7f206419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 0x7f206419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 -- 172.21.15.150:0/4285472886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064101050 msgr2=0x7f20641985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064101050 0x7f20641985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 -- 172.21.15.150:0/4285472886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206419d030 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f2063fff640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f205400e9f0 tx=0x7f205400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205400cd80 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206419d320 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2054005500 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054010690 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.325+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20641a4bb0 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.326+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2064101dd0 con 0x7f2064100690 2024-03-18T16:12:42.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.327+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f20540046c0 con 0x7f2064100690 2024-03-18T16:12:42.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.328+0000 7f20617fa640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 0x7f204c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.328+0000 7f20637fe640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 0x7f204c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.328+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f20540cf3c0 con 0x7f2064100690 2024-03-18T16:12:42.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.328+0000 7f20637fe640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 0x7f204c0a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20640ffe60 tx=0x7f2058005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.331+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2054053740 con 0x7f2064100690 2024-03-18T16:12:42.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.505+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2064199390 con 0x7f2064100690 2024-03-18T16:12:42.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.506+0000 7f20617fa640 1 -- 172.21.15.150:0/4285472886 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7f205409c120 con 0x7f2064100690 2024-03-18T16:12:42.507 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-18T16:12:42.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.507+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 msgr2=0x7f204c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.507+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 0x7f204c0a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20640ffe60 tx=0x7f2058005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.507+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 msgr2=0x7f2064198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.507+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f205400e9f0 tx=0x7f205400edf0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 shutdown_connections 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f204c0a2970 0x7f204c0a4e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20641066a0 0x7f206419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2064101050 0x7f20641985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 --2- 172.21.15.150:0/4285472886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2064100690 0x7f2064198090 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 >> 172.21.15.150:0/4285472886 conn(0x7f2064076a60 msgr2=0x7f20640fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 shutdown_connections 2024-03-18T16:12:42.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.508+0000 7f206a8f7640 1 -- 172.21.15.150:0/4285472886 wait complete. 2024-03-18T16:12:42.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 -- 172.21.15.150:0/2859873170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f702810be20 msgr2=0x7f702810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 --2- 172.21.15.150:0/2859873170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f702810be20 0x7f702810e290 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7020009ca0 tx=0x7f702002c180 comp rx=0 tx=0).stop 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 -- 172.21.15.150:0/2859873170 shutdown_connections 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 --2- 172.21.15.150:0/2859873170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f702810be20 0x7f702810e290 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 --2- 172.21.15.150:0/2859873170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7028103370 0x7f702810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 --2- 172.21.15.150:0/2859873170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 -- 172.21.15.150:0/2859873170 >> 172.21.15.150:0/2859873170 conn(0x7f7028077840 msgr2=0x7f7028077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 -- 172.21.15.150:0/2859873170 shutdown_connections 2024-03-18T16:12:42.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 -- 172.21.15.150:0/2859873170 wait complete. 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.624+0000 7f702c937640 1 Processor -- start 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 -- start start 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702810be20 0x7f702819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7028111bf0 con 0x7f7028103370 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7028111a70 con 0x7f702810be20 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7028111d70 con 0x7f7028102a50 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56398/0 (socket says 172.21.15.150:56398) 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 -- 172.21.15.150:0/1595843405 learned_addr learned my addr 172.21.15.150:0/1595843405 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702659c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7026d9d640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702810be20 0x7f702819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 -- 172.21.15.150:0/1595843405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 msgr2=0x7f7028198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 -- 172.21.15.150:0/1595843405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702810be20 msgr2=0x7f702819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702810be20 0x7f702819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 -- 172.21.15.150:0/1595843405 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702819d010 con 0x7f7028103370 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702659c640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f7025d9b640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e9f0 tx=0x7f701c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c00cd80 con 0x7f7028103370 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f701c010910 con 0x7f7028103370 2024-03-18T16:12:42.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c018990 con 0x7f7028103370 2024-03-18T16:12:42.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.625+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702819d300 con 0x7f7028103370 2024-03-18T16:12:42.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.626+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70281a4b90 con 0x7f7028103370 2024-03-18T16:12:42.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.627+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7028068a20 con 0x7f7028103370 2024-03-18T16:12:42.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.628+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f701c010ac0 con 0x7f7028103370 2024-03-18T16:12:42.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.628+0000 7f70177fe640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 0x7f70100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.628+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f701c0cf100 con 0x7f7028103370 2024-03-18T16:12:42.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.628+0000 7f702659c640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 0x7f70100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.629+0000 7f702659c640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 0x7f70100a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f70180095f0 tx=0x7f7018005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.633+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f701c09b810 con 0x7f7028103370 2024-03-18T16:12:42.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.810+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7028106f70 con 0x7f7028103370 2024-03-18T16:12:42.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.810+0000 7f70177fe640 1 -- 172.21.15.150:0/1595843405 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f701c09f8b0 con 0x7f7028103370 2024-03-18T16:12:42.812 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-18T16:12:42.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 msgr2=0x7f70100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 0x7f70100a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f70180095f0 tx=0x7f7018005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:42.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 msgr2=0x7f70281985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e9f0 tx=0x7f701c00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 shutdown_connections 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f70100a2970 0x7f70100a4e30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f702810be20 0x7f702819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7028103370 0x7f70281985b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 --2- 172.21.15.150:0/1595843405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7028102a50 0x7f7028198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 >> 172.21.15.150:0/1595843405 conn(0x7f7028077840 msgr2=0x7f70280fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 shutdown_connections 2024-03-18T16:12:42.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.812+0000 7f702c937640 1 -- 172.21.15.150:0/1595843405 wait complete. 2024-03-18T16:12:42.824 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:42 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1524999265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:42.824 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:42 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2104511545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:42.824 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:42 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/4285472886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:42.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:12:42.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 -- 172.21.15.150:0/1465656652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 msgr2=0x7f5b4010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 --2- 172.21.15.150:0/1465656652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b4010d5d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009cd0 tx=0x7f5b3802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 -- 172.21.15.150:0/1465656652 shutdown_connections 2024-03-18T16:12:42.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 --2- 172.21.15.150:0/1465656652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b4010d5d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 --2- 172.21.15.150:0/1465656652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40100700 0x7f5b40100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 --2- 172.21.15.150:0/1465656652 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40106710 0x7f5b40106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 -- 172.21.15.150:0/1465656652 >> 172.21.15.150:0/1465656652 conn(0x7f5b40077840 msgr2=0x7f5b40077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 -- 172.21.15.150:0/1465656652 shutdown_connections 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 -- 172.21.15.150:0/1465656652 wait complete. 2024-03-18T16:12:42.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.927+0000 7f5b460d1640 1 Processor -- start 2024-03-18T16:12:42.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 -- start start 2024-03-18T16:12:42.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40100700 0x7f5b40198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40106710 0x7f5b401a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40110b20 con 0x7f5b401010c0 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401109a0 con 0x7f5b40106710 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b460d1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40110ca0 con 0x7f5b40100700 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56422/0 (socket says 172.21.15.150:56422) 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 -- 172.21.15.150:0/393589048 learned_addr learned my addr 172.21.15.150:0/393589048 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3f7fe640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40100700 0x7f5b40198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3ffff640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40106710 0x7f5b401a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 -- 172.21.15.150:0/393589048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40100700 msgr2=0x7f5b40198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40100700 0x7f5b40198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 -- 172.21.15.150:0/393589048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40106710 msgr2=0x7f5b401a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40106710 0x7f5b401a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 -- 172.21.15.150:0/393589048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b401a44c0 con 0x7f5b401010c0 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.928+0000 7f5b3effd640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000e9f0 tx=0x7f5b3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.929+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3000cd80 con 0x7f5b401010c0 2024-03-18T16:12:42.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.929+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b30010910 con 0x7f5b401010c0 2024-03-18T16:12:42.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.929+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b401a4690 con 0x7f5b401010c0 2024-03-18T16:12:42.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.929+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30018990 con 0x7f5b401010c0 2024-03-18T16:12:42.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.929+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b401a4be0 con 0x7f5b401010c0 2024-03-18T16:12:42.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.930+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b40101e40 con 0x7f5b401010c0 2024-03-18T16:12:42.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.931+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f5b30010ac0 con 0x7f5b401010c0 2024-03-18T16:12:42.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.932+0000 7f5b3cff9640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 0x7f5b280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:42.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.932+0000 7f5b3f7fe640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 0x7f5b280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:42.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.932+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f5b300cf590 con 0x7f5b401010c0 2024-03-18T16:12:42.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.932+0000 7f5b3f7fe640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 0x7f5b280a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b340095f0 tx=0x7f5b34005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:42.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:42.935+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b3009bca0 con 0x7f5b401010c0 2024-03-18T16:12:43.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:42 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1524999265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:43.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:42 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2104511545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:43.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:42 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/4285472886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:43.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.123+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5b40108c70 con 0x7f5b401010c0 2024-03-18T16:12:43.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.123+0000 7f5b3cff9640 1 -- 172.21.15.150:0/393589048 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v103) ==== 81+0+1017 (secure 0 0 0) 0x7f5b3009fd40 con 0x7f5b401010c0 2024-03-18T16:12:43.125 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/1772653073 2024-03-19T15:50:29.895830+0000 2024-03-18T16:12:43.125 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/2222781423 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6801/2001484430 2024-03-19T15:50:29.895830+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6800/3095497318 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6800/2001484430 2024-03-19T15:50:29.895830+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6801/775411148 2024-03-19T15:50:55.359841+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/3872392632 2024-03-19T15:50:29.895830+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/2417146623 2024-03-19T15:50:55.359841+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/716777891 2024-03-19T15:50:55.359841+0000 2024-03-18T16:12:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/1896654966 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/3130851108 2024-03-18T17:04:57.464453+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/520847709 2024-03-19T15:50:55.359841+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6801/3095497318 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/2882492618 2024-03-19T15:50:29.895830+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/3143158665 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6800/775411148 2024-03-19T15:50:55.359841+0000 2024-03-18T16:12:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:0/3506843134 2024-03-19T16:00:04.468782+0000 2024-03-18T16:12:43.128 INFO:teuthology.orchestra.run.smithi150.stderr:listed 17 entries 2024-03-18T16:12:43.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 msgr2=0x7f5b280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 0x7f5b280a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b340095f0 tx=0x7f5b34005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 msgr2=0x7f5b40199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000e9f0 tx=0x7f5b3000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 shutdown_connections 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f5b280a2970 0x7f5b280a4e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b40106710 0x7f5b401a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b401010c0 0x7f5b40199440 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 --2- 172.21.15.150:0/393589048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b40100700 0x7f5b40198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 >> 172.21.15.150:0/393589048 conn(0x7f5b40077840 msgr2=0x7f5b401032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:43.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 shutdown_connections 2024-03-18T16:12:43.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.125+0000 7f5b460d1640 1 -- 172.21.15.150:0/393589048 wait complete. 2024-03-18T16:12:43.138 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T16:12:43.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:43.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T16:12:43.211 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T16:12:43.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:43.211 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T16:12:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:42 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1524999265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:42 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2104511545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:12:43.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:42 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/4285472886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:12:43.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:43.255 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-18T16:12:43.323 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T16:12:43.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:43.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T16:12:43.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:43.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-03-18T16:12:43.499 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T16:12:43.509 INFO:tasks.cephadm:Teardown begin 2024-03-18T16:12:43.510 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T16:12:43.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:12:43.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 -- 172.21.15.150:0/1709184062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0102130 msgr2=0x7f9ef010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/1709184062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0102130 0x7f9ef010e640 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009cd0 tx=0x7f9ee802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 -- 172.21.15.150:0/1709184062 shutdown_connections 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/1709184062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0102130 0x7f9ef010e640 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/1709184062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0101770 0x7f9ef0101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/1709184062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0107780 0x7f9ef0107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 -- 172.21.15.150:0/1709184062 >> 172.21.15.150:0/1709184062 conn(0x7f9ef00fd280 msgr2=0x7f9ef00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 -- 172.21.15.150:0/1709184062 shutdown_connections 2024-03-18T16:12:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.613+0000 7f9ef87b9640 1 -- 172.21.15.150:0/1709184062 wait complete. 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 Processor -- start 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 -- start start 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0101770 0x7f9ef0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0102130 0x7f9ef01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0111b90 con 0x7f9ef0107780 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0111a10 con 0x7f9ef0101770 2024-03-18T16:12:43.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef87b9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0111d10 con 0x7f9ef0102130 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56446/0 (socket says 172.21.15.150:56446) 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 -- 172.21.15.150:0/648695600 learned_addr learned my addr 172.21.15.150:0/648695600 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef5d54640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0102130 0x7f9ef01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6555640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0101770 0x7f9ef0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 -- 172.21.15.150:0/648695600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0102130 msgr2=0x7f9ef01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0102130 0x7f9ef01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 -- 172.21.15.150:0/648695600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0101770 msgr2=0x7f9ef0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0101770 0x7f9ef0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 -- 172.21.15.150:0/648695600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef019d060 con 0x7f9ef0107780 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.614+0000 7f9ef6d56640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80348f0 tx=0x7f9ee8032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.615+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8041020 con 0x7f9ef0107780 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.615+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef019d2f0 con 0x7f9ef0107780 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.615+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee8032e80 con 0x7f9ef0107780 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.615+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8039040 con 0x7f9ef0107780 2024-03-18T16:12:43.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.615+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef01a4be0 con 0x7f9ef0107780 2024-03-18T16:12:43.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.616+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef0068a20 con 0x7f9ef0107780 2024-03-18T16:12:43.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.617+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f9ee8044020 con 0x7f9ef0107780 2024-03-18T16:12:43.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.617+0000 7f9ee77fe640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 0x7f9edc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.617+0000 7f9ef6555640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 0x7f9edc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.617+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f9ee80f0d10 con 0x7f9ef0107780 2024-03-18T16:12:43.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.618+0000 7f9ef6555640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 0x7f9edc0a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009b50 tx=0x7f9eec005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:43.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.621+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ee804b020 con 0x7f9ef0107780 2024-03-18T16:12:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.838+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ef01993c0 con 0x7f9ef0107780 2024-03-18T16:12:43.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.839+0000 7f9ee77fe640 1 -- 172.21.15.150:0/648695600 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11885 (secure 0 0 0) 0x7f9ef01993c0 con 0x7f9ef0107780 2024-03-18T16:12:43.841 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:43.841 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":15930,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/851431923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":851431923},{"type":"v1","addr":"172.21.15.191:6835","nonce":851431923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T16:06:17.634932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24496,24481,24485]},"id":1}]} 2024-03-18T16:12:43.841 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 msgr2=0x7f9edc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 0x7f9edc0a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009b50 tx=0x7f9eec005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 msgr2=0x7f9ef019c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80348f0 tx=0x7f9ee8032c90 comp rx=0 tx=0).stop 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 shutdown_connections 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f9edc0a2970 0x7f9edc0a4e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef0107780 0x7f9ef019c930 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ef0102130 0x7f9ef01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 --2- 172.21.15.150:0/648695600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ef0101770 0x7f9ef0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 >> 172.21.15.150:0/648695600 conn(0x7f9ef00fd280 msgr2=0x7f9ef00fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:43.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 shutdown_connections 2024-03-18T16:12:43.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.841+0000 7f9ef87b9640 1 -- 172.21.15.150:0/648695600 wait complete. 2024-03-18T16:12:43.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:12:43.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 -- 172.21.15.150:0/4235273705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa406a830 msgr2=0x7fdfa410e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/4235273705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa406a830 0x7fdfa410e630 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009ca0 tx=0x7fdf9c02c180 comp rx=0 tx=0).stop 2024-03-18T16:12:43.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 -- 172.21.15.150:0/4235273705 shutdown_connections 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/4235273705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa406a830 0x7fdfa410e630 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/4235273705 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa406a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/4235273705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa41077e0 0x7fdfa4107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 -- 172.21.15.150:0/4235273705 >> 172.21.15.150:0/4235273705 conn(0x7fdfa4077780 msgr2=0x7fdfa4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 -- 172.21.15.150:0/4235273705 shutdown_connections 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.954+0000 7fdfaa33b640 1 -- 172.21.15.150:0/4235273705 wait complete. 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 Processor -- start 2024-03-18T16:12:43.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 -- start start 2024-03-18T16:12:43.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa406a830 0x7fdfa4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa4111ba0 con 0x7fdfa41077e0 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa4111a20 con 0x7fdfa406a830 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfaa33b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa4111d20 con 0x7fdfa4069e90 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56454/0 (socket says 172.21.15.150:56454) 2024-03-18T16:12:43.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 -- 172.21.15.150:0/2970857322 learned_addr learned my addr 172.21.15.150:0/2970857322 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa37fe640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa406a830 0x7fdfa4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 -- 172.21.15.150:0/2970857322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 msgr2=0x7fdfa4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 -- 172.21.15.150:0/2970857322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa406a830 msgr2=0x7fdfa4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.955+0000 7fdfa88d8640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa406a830 0x7fdfa4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa88d8640 1 -- 172.21.15.150:0/2970857322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa419cfd0 con 0x7fdfa41077e0 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa3fff640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa88d8640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02c790 tx=0x7fdf9c035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:43.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c039040 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c044070 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa419d260 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c032940 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.956+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa41a4b50 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.957+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa4102f00 con 0x7fdfa41077e0 2024-03-18T16:12:43.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.958+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7fdf9c032ae0 con 0x7fdfa41077e0 2024-03-18T16:12:43.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.958+0000 7fdfa17fa640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 0x7fdf8c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:43.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.959+0000 7fdfa3fff640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 0x7fdf8c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:43.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.959+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7fdf9c0f11d0 con 0x7fdfa41077e0 2024-03-18T16:12:43.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.959+0000 7fdfa3fff640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 0x7fdf8c0a4eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009f40 tx=0x7fdf98005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:43.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:43.963+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf9c0be070 con 0x7fdfa41077e0 2024-03-18T16:12:44.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:43 smithi156 ceph-mon[35919]: pgmap v396: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:43 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1595843405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:44.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:43 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/393589048' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:12:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:43 smithi150 ceph-mon[31946]: pgmap v396: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:43 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1595843405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:44.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:43 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/393589048' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:12:44.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.177+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfa4109c80 con 0x7fdfa41077e0 2024-03-18T16:12:44.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.178+0000 7fdfa17fa640 1 -- 172.21.15.150:0/2970857322 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11885 (secure 0 0 0) 0x7fdf9c0bd970 con 0x7fdfa41077e0 2024-03-18T16:12:44.180 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:44.180 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3457359826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3457359826},{"type":"v1","addr":"172.21.15.150:6835","nonce":3457359826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3172588678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3172588678},{"type":"v1","addr":"172.21.15.150:6837","nonce":3172588678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/4258774828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":4258774828},{"type":"v1","addr":"172.21.15.150:6839","nonce":4258774828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6841/4215738622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6840","nonce":4215738622},{"type":"v1","addr":"172.21.15.150:6841","nonce":4215738622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":15930,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/851431923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":851431923},{"type":"v1","addr":"172.21.15.191:6835","nonce":851431923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/925824611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":925824611},{"type":"v1","addr":"172.21.15.156:6833","nonce":925824611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T15:57:56.512066+0000","modified":"2024-03-18T16:06:17.634932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24496,"mds_1":24485,"mds_2":24481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/4268627742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":4268627742},{"type":"v1","addr":"172.21.15.156:6835","nonce":4268627742}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/2692248421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2692248421},{"type":"v1","addr":"172.21.15.191:6833","nonce":2692248421}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/1894652569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1894652569},{"type":"v1","addr":"172.21.15.156:6837","nonce":1894652569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/720633938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":720633938},{"type":"v1","addr":"172.21.15.191:6837","nonce":720633938}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":1,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.156:6839/219054917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":219054917},{"type":"v1","addr":"172.21.15.156:6839","nonce":219054917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/3948564183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":3948564183},{"type":"v1","addr":"172.21.15.191:6839","nonce":3948564183}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24496,"qdb_cluster":[24496,24481,24485]},"id":1}]} 2024-03-18T16:12:44.180 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 msgr2=0x7fdf8c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 0x7fdf8c0a4eb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009f40 tx=0x7fdf98005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 msgr2=0x7fdfa419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02c790 tx=0x7fdf9c035e00 comp rx=0 tx=0).stop 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 shutdown_connections 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7fdf8c0a29f0 0x7fdf8c0a4eb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa41077e0 0x7fdfa419c8a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfa406a830 0x7fdfa4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 --2- 172.21.15.150:0/2970857322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfa4069e90 0x7fdfa4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 >> 172.21.15.150:0/2970857322 conn(0x7fdfa4077780 msgr2=0x7fdfa40ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 shutdown_connections 2024-03-18T16:12:44.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.180+0000 7fdfaa33b640 1 -- 172.21.15.150:0/2970857322 wait complete. 2024-03-18T16:12:44.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:12:44.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:43 smithi191 ceph-mon[36784]: pgmap v396: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:44.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:43 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1595843405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:12:44.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:43 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/393589048' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:12:44.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 -- 172.21.15.150:0/2182820553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0069ed0 msgr2=0x7f6be006a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 --2- 172.21.15.150:0/2182820553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0069ed0 0x7f6be006a330 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009900 tx=0x7f6bd402bcd0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 -- 172.21.15.150:0/2182820553 shutdown_connections 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 --2- 172.21.15.150:0/2182820553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be006a870 0x7f6be010e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 --2- 172.21.15.150:0/2182820553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0069ed0 0x7f6be006a330 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 --2- 172.21.15.150:0/2182820553 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be01077e0 0x7f6be0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 -- 172.21.15.150:0/2182820553 >> 172.21.15.150:0/2182820553 conn(0x7f6be0077780 msgr2=0x7f6be0077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 -- 172.21.15.150:0/2182820553 shutdown_connections 2024-03-18T16:12:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.293+0000 7f6be51b9640 1 -- 172.21.15.150:0/2182820553 wait complete. 2024-03-18T16:12:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 Processor -- start 2024-03-18T16:12:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 -- start start 2024-03-18T16:12:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be0069ed0 0x7f6be0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 0x7f6be019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6be0111ba0 con 0x7f6be01077e0 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6be0111a20 con 0x7f6be0069ed0 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.294+0000 7f6be51b9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f6be0111d20 con 0x7f6be006a870 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bdf59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 0x7f6be019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bded9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be0069ed0 0x7f6be0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.150:50932/0 (socket says 172.21.15.150:50932) 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bdf59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 0x7f6be019c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56480/0 (socket says 172.21.15.150:56480) 2024-03-18T16:12:44.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 -- 172.21.15.150:0/1952976975 learned_addr learned my addr 172.21.15.150:0/1952976975 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 -- 172.21.15.150:0/1952976975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be0069ed0 msgr2=0x7f6be0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be0069ed0 0x7f6be0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 -- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 msgr2=0x7f6be019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 0x7f6be019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 -- 172.21.15.150:0/1952976975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be019d070 con 0x7f6be006a870 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bde59c640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4005940 tx=0x7f6bd402cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4041070 con 0x7f6be006a870 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be019d300 con 0x7f6be006a870 2024-03-18T16:12:44.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd4004450 con 0x7f6be006a870 2024-03-18T16:12:44.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003df0 con 0x7f6be006a870 2024-03-18T16:12:44.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.295+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be01a4bf0 con 0x7f6be006a870 2024-03-18T16:12:44.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.296+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be0102f00 con 0x7f6be006a870 2024-03-18T16:12:44.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.297+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f6bd4044070 con 0x7f6be006a870 2024-03-18T16:12:44.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.298+0000 7f6bc7fff640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 0x7f6bc80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.298+0000 7f6bded9d640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 0x7f6bc80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.298+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f6bd40f0a60 con 0x7f6be006a870 2024-03-18T16:12:44.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.298+0000 7f6bded9d640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 0x7f6bc80a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009b50 tx=0x7f6bd0005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.301+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6bd40bd280 con 0x7f6be006a870 2024-03-18T16:12:44.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.477+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6be0109c80 con 0x7f6be006a870 2024-03-18T16:12:44.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.479+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1952976975 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+23202 (secure 0 0 0) 0x7f6bd40c1320 con 0x7f6be006a870 2024-03-18T16:12:44.481 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-18T16:12:44.481 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":103,"fsid":"0cd032e0-e53f-11ee-95c9-87774f69a715","created":"2024-03-18T15:49:59.320410+0000","modified":"2024-03-18T16:12:09.425191+0000","last_up_change":"2024-03-18T15:57:06.231086+0000","last_in_change":"2024-03-18T15:56:46.311207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T15:53:17.702512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T15:57:54.584892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T15:57:55.700788+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"34d33d5d-faae-42ac-b7d3-70679106ee85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6803","nonce":4075395622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6805","nonce":4075395622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6809","nonce":4075395622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4075395622},{"type":"v1","addr":"172.21.15.150:6807","nonce":4075395622}]},"public_addr":"172.21.15.150:6803/4075395622","cluster_addr":"172.21.15.150:6805/4075395622","heartbeat_back_addr":"172.21.15.150:6809/4075395622","heartbeat_front_addr":"172.21.15.150:6807/4075395622","state":["exists","up"]},{"osd":1,"uuid":"14890f1b-f4e5-4d2d-a1f0-71c84f464fd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6801","nonce":737563874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6803","nonce":737563874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6807","nonce":737563874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":737563874},{"type":"v1","addr":"172.21.15.156:6805","nonce":737563874}]},"public_addr":"172.21.15.156:6801/737563874","cluster_addr":"172.21.15.156:6803/737563874","heartbeat_back_addr":"172.21.15.156:6807/737563874","heartbeat_front_addr":"172.21.15.156:6805/737563874","state":["exists","up"]},{"osd":2,"uuid":"1f5a563a-5180-40bd-a2a9-625033da15e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6801","nonce":1228471627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6803","nonce":1228471627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6807","nonce":1228471627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1228471627},{"type":"v1","addr":"172.21.15.191:6805","nonce":1228471627}]},"public_addr":"172.21.15.191:6801/1228471627","cluster_addr":"172.21.15.191:6803/1228471627","heartbeat_back_addr":"172.21.15.191:6807/1228471627","heartbeat_front_addr":"172.21.15.191:6805/1228471627","state":["exists","up"]},{"osd":3,"uuid":"869ada62-adef-48b5-8d36-f57fb0220c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6811","nonce":2684549380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6813","nonce":2684549380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6817","nonce":2684549380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2684549380},{"type":"v1","addr":"172.21.15.150:6815","nonce":2684549380}]},"public_addr":"172.21.15.150:6811/2684549380","cluster_addr":"172.21.15.150:6813/2684549380","heartbeat_back_addr":"172.21.15.150:6817/2684549380","heartbeat_front_addr":"172.21.15.150:6815/2684549380","state":["exists","up"]},{"osd":4,"uuid":"7324a8d6-300d-4163-8282-c368870e2bfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6809","nonce":1341344741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6811","nonce":1341344741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6815","nonce":1341344741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1341344741},{"type":"v1","addr":"172.21.15.156:6813","nonce":1341344741}]},"public_addr":"172.21.15.156:6809/1341344741","cluster_addr":"172.21.15.156:6811/1341344741","heartbeat_back_addr":"172.21.15.156:6815/1341344741","heartbeat_front_addr":"172.21.15.156:6813/1341344741","state":["exists","up"]},{"osd":5,"uuid":"53bfde8f-b1e3-4482-ac6a-e17c6a095ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6809","nonce":3189109632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6811","nonce":3189109632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6815","nonce":3189109632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3189109632},{"type":"v1","addr":"172.21.15.191:6813","nonce":3189109632}]},"public_addr":"172.21.15.191:6809/3189109632","cluster_addr":"172.21.15.191:6811/3189109632","heartbeat_back_addr":"172.21.15.191:6815/3189109632","heartbeat_front_addr":"172.21.15.191:6813/3189109632","state":["exists","up"]},{"osd":6,"uuid":"7735e727-557c-4104-9d72-5157f53ec694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6819","nonce":4061900525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6821","nonce":4061900525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6825","nonce":4061900525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":4061900525},{"type":"v1","addr":"172.21.15.150:6823","nonce":4061900525}]},"public_addr":"172.21.15.150:6819/4061900525","cluster_addr":"172.21.15.150:6821/4061900525","heartbeat_back_addr":"172.21.15.150:6825/4061900525","heartbeat_front_addr":"172.21.15.150:6823/4061900525","state":["exists","up"]},{"osd":7,"uuid":"ce254a57-8b1c-4b10-b76e-510130e8d399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6817","nonce":255625842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6819","nonce":255625842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6823","nonce":255625842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":255625842},{"type":"v1","addr":"172.21.15.156:6821","nonce":255625842}]},"public_addr":"172.21.15.156:6817/255625842","cluster_addr":"172.21.15.156:6819/255625842","heartbeat_back_addr":"172.21.15.156:6823/255625842","heartbeat_front_addr":"172.21.15.156:6821/255625842","state":["exists","up"]},{"osd":8,"uuid":"aa202baf-1dea-495c-9c8a-637cbbba34c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6817","nonce":2388059353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6819","nonce":2388059353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6823","nonce":2388059353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2388059353},{"type":"v1","addr":"172.21.15.191:6821","nonce":2388059353}]},"public_addr":"172.21.15.191:6817/2388059353","cluster_addr":"172.21.15.191:6819/2388059353","heartbeat_back_addr":"172.21.15.191:6823/2388059353","heartbeat_front_addr":"172.21.15.191:6821/2388059353","state":["exists","up"]},{"osd":9,"uuid":"e301a14e-5738-44bf-af14-269a3be744dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6827","nonce":145936256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6829","nonce":145936256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6833","nonce":145936256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":145936256},{"type":"v1","addr":"172.21.15.150:6831","nonce":145936256}]},"public_addr":"172.21.15.150:6827/145936256","cluster_addr":"172.21.15.150:6829/145936256","heartbeat_back_addr":"172.21.15.150:6833/145936256","heartbeat_front_addr":"172.21.15.150:6831/145936256","state":["exists","up"]},{"osd":10,"uuid":"1f122516-01f0-41e8-9ff8-c73bbf6609d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6825","nonce":4270466180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6827","nonce":4270466180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6831","nonce":4270466180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4270466180},{"type":"v1","addr":"172.21.15.156:6829","nonce":4270466180}]},"public_addr":"172.21.15.156:6825/4270466180","cluster_addr":"172.21.15.156:6827/4270466180","heartbeat_back_addr":"172.21.15.156:6831/4270466180","heartbeat_front_addr":"172.21.15.156:6829/4270466180","state":["exists","up"]},{"osd":11,"uuid":"0691d334-c620-436b-b0f8-8343b4c4cf33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6825","nonce":2841370319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6827","nonce":2841370319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6831","nonce":2841370319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2841370319},{"type":"v1","addr":"172.21.15.191:6829","nonce":2841370319}]},"public_addr":"172.21.15.191:6825/2841370319","cluster_addr":"172.21.15.191:6827/2841370319","heartbeat_back_addr":"172.21.15.191:6831/2841370319","heartbeat_front_addr":"172.21.15.191:6829/2841370319","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:28.650275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:52:51.617206+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:15.071610+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:53:39.957483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:03.890011+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:29.632268+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:54:54.907700+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:20.015762+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:55:46.392050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:12.741895+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:56:38.341634+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T15:57:04.977039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:0/1772653073":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2222781423":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6801/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6800/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/2001484430":"2024-03-19T15:50:29.895830+0000","172.21.15.150:6801/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3872392632":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/2417146623":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/716777891":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/1896654966":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/3130851108":"2024-03-18T17:04:57.464453+0000","172.21.15.150:0/520847709":"2024-03-19T15:50:55.359841+0000","172.21.15.150:6801/3095497318":"2024-03-19T16:00:04.468782+0000","172.21.15.150:0/2882492618":"2024-03-19T15:50:29.895830+0000","172.21.15.150:0/3143158665":"2024-03-19T16:00:04.468782+0000","172.21.15.150:6800/775411148":"2024-03-19T15:50:55.359841+0000","172.21.15.150:0/3506843134":"2024-03-19T16:00:04.468782+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 msgr2=0x7f6bc80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 0x7f6bc80a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009b50 tx=0x7f6bd0005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 msgr2=0x7f6be01985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4005940 tx=0x7f6bd402cc40 comp rx=0 tx=0).stop 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 shutdown_connections 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f6bc80a2970 0x7f6bc80a4e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be01077e0 0x7f6be019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6be006a870 0x7f6be01985c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 --2- 172.21.15.150:0/1952976975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6be0069ed0 0x7f6be0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 >> 172.21.15.150:0/1952976975 conn(0x7f6be0077780 msgr2=0x7f6be00ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 shutdown_connections 2024-03-18T16:12:44.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.481+0000 7f6be51b9640 1 -- 172.21.15.150:0/1952976975 wait complete. 2024-03-18T16:12:44.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- 172.21.15.150:0/1063099339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14106660 msgr2=0x7f3e14106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 --2- 172.21.15.150:0/1063099339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14106660 0x7f3e14106a40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009930 tx=0x7f3e0802bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- 172.21.15.150:0/1063099339 shutdown_connections 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 --2- 172.21.15.150:0/1063099339 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14101010 0x7f3e1410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 --2- 172.21.15.150:0/1063099339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14100650 0x7f3e14100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 --2- 172.21.15.150:0/1063099339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14106660 0x7f3e14106a40 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- 172.21.15.150:0/1063099339 >> 172.21.15.150:0/1063099339 conn(0x7f3e140777e0 msgr2=0x7f3e14077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- 172.21.15.150:0/1063099339 shutdown_connections 2024-03-18T16:12:44.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- 172.21.15.150:0/1063099339 wait complete. 2024-03-18T16:12:44.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 Processor -- start 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.596+0000 7f3e197e8640 1 -- start start 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 0x7f3e14198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14106660 0x7f3e1419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e14111b70 con 0x7f3e14101010 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e141119f0 con 0x7f3e14106660 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f3e14111cf0 con 0x7f3e14100650 2024-03-18T16:12:44.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56498/0 (socket says 172.21.15.150:56498) 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 -- 172.21.15.150:0/3752840777 learned_addr learned my addr 172.21.15.150:0/3752840777 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e12ffd640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 0x7f3e14198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e137fe640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14106660 0x7f3e1419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 -- 172.21.15.150:0/3752840777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 msgr2=0x7f3e14198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 0x7f3e14198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 -- 172.21.15.150:0/3752840777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14106660 msgr2=0x7f3e1419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14106660 0x7f3e1419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 -- 172.21.15.150:0/3752840777 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1419d080 con 0x7f3e14101010 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e12ffd640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 0x7f3e14198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e127fc640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e9f0 tx=0x7f3e0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0400cd80 con 0x7f3e14101010 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1419d370 con 0x7f3e14101010 2024-03-18T16:12:44.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e04010910 con 0x7f3e14101010 2024-03-18T16:12:44.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.597+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e040188f0 con 0x7f3e14101010 2024-03-18T16:12:44.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.598+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e141a4c00 con 0x7f3e14101010 2024-03-18T16:12:44.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.598+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e140685d0 con 0x7f3e14101010 2024-03-18T16:12:44.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.600+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f3e04010ac0 con 0x7f3e14101010 2024-03-18T16:12:44.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.600+0000 7f3dfbfff640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 0x7f3dfc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.600+0000 7f3e12ffd640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 0x7f3dfc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.600+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8527+0+0 (secure 0 0 0) 0x7f3e040cf5e0 con 0x7f3e14101010 2024-03-18T16:12:44.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.601+0000 7f3e12ffd640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 0x7f3dfc0a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3e08005ab0 tx=0x7f3e08005fd0 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.604+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e0409bcf0 con 0x7f3e14101010 2024-03-18T16:12:44.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.816+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3e141993e0 con 0x7f3e14101010 2024-03-18T16:12:44.820 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:44 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/648695600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:44.820 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:44 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/2970857322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:44.820 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:44 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1952976975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:44.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.820+0000 7f3dfbfff640 1 -- 172.21.15.150:0/3752840777 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7f3e141993e0 con 0x7f3e14101010 2024-03-18T16:12:44.822 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T16:12:44.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 msgr2=0x7f3dfc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 0x7f3dfc0a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3e08005ab0 tx=0x7f3e08005fd0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 msgr2=0x7f3e141985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e9f0 tx=0x7f3e0400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 shutdown_connections 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f3dfc0a2970 0x7f3dfc0a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e14106660 0x7f3e1419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e14101010 0x7f3e141985d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 --2- 172.21.15.150:0/3752840777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e14100650 0x7f3e14198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 >> 172.21.15.150:0/3752840777 conn(0x7f3e140777e0 msgr2=0x7f3e14102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 shutdown_connections 2024-03-18T16:12:44.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.823+0000 7f3e197e8640 1 -- 172.21.15.150:0/3752840777 wait complete. 2024-03-18T16:12:44.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 -- 172.21.15.150:0/3006003128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102130 msgr2=0x7f43b010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 --2- 172.21.15.150:0/3006003128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102130 0x7f43b010e640 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009cd0 tx=0x7f43a802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 -- 172.21.15.150:0/3006003128 shutdown_connections 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 --2- 172.21.15.150:0/3006003128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0102130 0x7f43b010e640 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 --2- 172.21.15.150:0/3006003128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 --2- 172.21.15.150:0/3006003128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0107780 0x7f43b0107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 -- 172.21.15.150:0/3006003128 >> 172.21.15.150:0/3006003128 conn(0x7f43b0078be0 msgr2=0x7f43b0078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 -- 172.21.15.150:0/3006003128 shutdown_connections 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.943+0000 7f43b7959640 1 -- 172.21.15.150:0/3006003128 wait complete. 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 Processor -- start 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 -- start start 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0102130 0x7f43b0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0111b90 con 0x7f43b0107780 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0111a10 con 0x7f43b0101770 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b7959640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0111d10 con 0x7f43b0102130 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56518/0 (socket says 172.21.15.150:56518) 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 -- 172.21.15.150:0/1685563666 learned_addr learned my addr 172.21.15.150:0/1685563666 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b56f5640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b4ef4640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0102130 0x7f43b0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 -- 172.21.15.150:0/1685563666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0102130 msgr2=0x7f43b0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0102130 0x7f43b0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 -- 172.21.15.150:0/1685563666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 msgr2=0x7f43b0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.944+0000 7f43b5ef6640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43b5ef6640 1 -- 172.21.15.150:0/1685563666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b019d040 con 0x7f43b0107780 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43b56f5640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43b5ef6640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f43a8004af0 tx=0x7f43a8032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8041020 con 0x7f43b0107780 2024-03-18T16:12:44.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43a8034cd0 con 0x7f43b0107780 2024-03-18T16:12:44.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b019d2d0 con 0x7f43b0107780 2024-03-18T16:12:44.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8039040 con 0x7f43b0107780 2024-03-18T16:12:44.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.945+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b01a4bc0 con 0x7f43b0107780 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.946+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b0068a20 con 0x7f43b0107780 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.947+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f43a8044020 con 0x7f43b0107780 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.948+0000 7f43a67fc640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 0x7f439c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.948+0000 7f43b56f5640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 0x7f439c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.948+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8741+0+0 (secure 0 0 0) 0x7f43a80f0f20 con 0x7f43b0107780 2024-03-18T16:12:44.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.948+0000 7f43b56f5640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 0x7f439c0a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43ac005fd0 tx=0x7f43ac00b390 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:44.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:44.951+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43a804b020 con 0x7f43b0107780 2024-03-18T16:12:45.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/648695600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/2970857322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1952976975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.019 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.019 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.019 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mds[66371]: -- 172.21.15.156:0/1281389244 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fdf644da40 con 0x55fdf6420c00 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: starting mds.b at 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mds[65740]: -- 172.21.15.156:0/2489275341 <== mon.0 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562eff755a40 con 0x562eff729000 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: starting mds.e at 2024-03-18T16:12:45.020 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.021 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.021 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-mds[66049]: -- 172.21.15.156:0/3083485027 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e126f3a40 con 0x563e126c7000 2024-03-18T16:12:45.021 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:12:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: starting mds.h at 2024-03-18T16:12:45.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a[31942]: 2024-03-18T16:12:44.817+0000 7f2f014c7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:12:45.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.164+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f43b0109ce0 con 0x7f43b0107780 2024-03-18T16:12:45.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/648695600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/2970857322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1952976975' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:12:45.213 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.213 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-mds[67939]: -- 172.21.15.191:0/1667038858 <== mon.0 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561598869a40 con 0x56159883cc00 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: starting mds.i at 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-mds[67629]: -- 172.21.15.191:0/178041019 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563079323a40 con 0x5630792f7000 2024-03-18T16:12:45.214 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: starting mds.c at 2024-03-18T16:12:45.215 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: ignoring --setuser ceph since I am not root 2024-03-18T16:12:45.215 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: ignoring --setgroup ceph since I am not root 2024-03-18T16:12:45.215 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:12:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: starting mds.l at 2024-03-18T16:12:45.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.821+0000 7f43a67fc640 1 -- 172.21.15.150:0/1685563666 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f43a80bddd0 con 0x7f43b0107780 2024-03-18T16:12:45.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 msgr2=0x7f439c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:45.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 0x7f439c0a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43ac005fd0 tx=0x7f43ac00b390 comp rx=0 tx=0).stop 2024-03-18T16:12:45.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 msgr2=0x7f43b019c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:45.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f43a8004af0 tx=0x7f43a8032ec0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 shutdown_connections 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f439c0a2970 0x7f439c0a4e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0107780 0x7f43b019c910 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43b0102130 0x7f43b0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 --2- 172.21.15.150:0/1685563666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43b0101770 0x7f43b0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 >> 172.21.15.150:0/1685563666 conn(0x7f43b0078be0 msgr2=0x7f43b0104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 shutdown_connections 2024-03-18T16:12:45.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.824+0000 7f43b7959640 1 -- 172.21.15.150:0/1685563666 wait complete. 2024-03-18T16:12:45.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T16:12:45.863 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: pgmap v397: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T16:12:45.864 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:45 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:12:45.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 -- 172.21.15.150:0/2311073727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c100660 msgr2=0x7f525c100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:45.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 --2- 172.21.15.150:0/2311073727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c100660 0x7f525c100ae0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5250009930 tx=0x7f525002bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:45.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 -- 172.21.15.150:0/2311073727 shutdown_connections 2024-03-18T16:12:45.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 --2- 172.21.15.150:0/2311073727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c101020 0x7f525c10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 --2- 172.21.15.150:0/2311073727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c100660 0x7f525c100ae0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 --2- 172.21.15.150:0/2311073727 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c106670 0x7f525c106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 -- 172.21.15.150:0/2311073727 >> 172.21.15.150:0/2311073727 conn(0x7f525c077890 msgr2=0x7f525c077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 -- 172.21.15.150:0/2311073727 shutdown_connections 2024-03-18T16:12:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.946+0000 7f5260859640 1 -- 172.21.15.150:0/2311073727 wait complete. 2024-03-18T16:12:45.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 Processor -- start 2024-03-18T16:12:45.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 -- start start 2024-03-18T16:12:45.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c100660 0x7f525c196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c106670 0x7f525c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f525c110a80 con 0x7f525c101020 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f525c110900 con 0x7f525c106670 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5260859640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f525c110c00 con 0x7f525c100660 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56532/0 (socket says 172.21.15.150:56532) 2024-03-18T16:12:45.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 -- 172.21.15.150:0/3950286076 learned_addr learned my addr 172.21.15.150:0/3950286076 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f525a59c640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c100660 0x7f525c196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f525ad9d640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c106670 0x7f525c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 -- 172.21.15.150:0/3950286076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c100660 msgr2=0x7f525c196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c100660 0x7f525c196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 -- 172.21.15.150:0/3950286076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c106670 msgr2=0x7f525c19b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.947+0000 7f5259d9b640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c106670 0x7f525c19b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f5259d9b640 1 -- 172.21.15.150:0/3950286076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c19bf90 con 0x7f525c101020 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f5259d9b640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5250009230 tx=0x7f525002cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:45.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250041070 con 0x7f525c101020 2024-03-18T16:12:45.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c19c220 con 0x7f525c101020 2024-03-18T16:12:45.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5250004450 con 0x7f525c101020 2024-03-18T16:12:45.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250034440 con 0x7f525c101020 2024-03-18T16:12:45.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.948+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c1a3b10 con 0x7f525c101020 2024-03-18T16:12:45.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.949+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c0685d0 con 0x7f525c101020 2024-03-18T16:12:45.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.950+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f5250044070 con 0x7f525c101020 2024-03-18T16:12:45.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.950+0000 7f524b7fe640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 0x7f52440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:45.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.951+0000 7f525a59c640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 0x7f52440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:45.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.951+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8741+0+0 (secure 0 0 0) 0x7f52500f07f0 con 0x7f525c101020 2024-03-18T16:12:45.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.951+0000 7f525a59c640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 0x7f52440a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f524c009b50 tx=0x7f524c005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:45.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:45.955+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52500bcf30 con 0x7f525c101020 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: pgmap v397: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:12:46.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T16:12:46.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T16:12:46.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:45 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:12:46.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.137+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f525c1982f0 con 0x7f525c101020 2024-03-18T16:12:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: pgmap v397: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 504 KiB/s wr, 21 op/s 2024-03-18T16:12:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:12:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:12:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:12:46.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3752840777' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:12:46.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: osdmap e104: 12 total, 12 up, 12 in 2024-03-18T16:12:46.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-18T16:12:46.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:45 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:12:46.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.826+0000 7f524b7fe640 1 -- 172.21.15.150:0/3950286076 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7f52500c0fd0 con 0x7f525c101020 2024-03-18T16:12:46.828 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-03-18T16:12:46.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 msgr2=0x7f52440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 0x7f52440a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f524c009b50 tx=0x7f524c005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 msgr2=0x7f525c1974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5250009230 tx=0x7f525002cba0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 shutdown_connections 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f52440a2970 0x7f52440a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c106670 0x7f525c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c101020 0x7f525c1974e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 --2- 172.21.15.150:0/3950286076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f525c100660 0x7f525c196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 >> 172.21.15.150:0/3950286076 conn(0x7f525c077890 msgr2=0x7f525c103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:46.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 shutdown_connections 2024-03-18T16:12:46.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.829+0000 7f5260859640 1 -- 172.21.15.150:0/3950286076 wait complete. 2024-03-18T16:12:46.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T16:12:46.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 -- 172.21.15.150:0/1292903336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974101770 msgr2=0x7f7974101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:46.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 --2- 172.21.15.150:0/1292903336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974101770 0x7f7974101bf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7968009930 tx=0x7f796802bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 -- 172.21.15.150:0/1292903336 shutdown_connections 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 --2- 172.21.15.150:0/1292903336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974102130 0x7f797410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 --2- 172.21.15.150:0/1292903336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974101770 0x7f7974101bf0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 --2- 172.21.15.150:0/1292903336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f7974107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 -- 172.21.15.150:0/1292903336 >> 172.21.15.150:0/1292903336 conn(0x7f7974078be0 msgr2=0x7f7974078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 -- 172.21.15.150:0/1292903336 shutdown_connections 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.994+0000 7f7979ee8640 1 -- 172.21.15.150:0/1292903336 wait complete. 2024-03-18T16:12:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 Processor -- start 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 -- start start 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974101770 0x7f7974198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f797419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7974111b90 con 0x7f7974102130 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7974111a10 con 0x7f7974107780 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7979ee8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 -- 0x7f7974111d10 con 0x7f7974101770 2024-03-18T16:12:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7972ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7972ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56548/0 (socket says 172.21.15.150:56548) 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f7972ffd640 1 -- 172.21.15.150:0/545681307 learned_addr learned my addr 172.21.15.150:0/545681307 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.995+0000 7f79737fe640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974101770 0x7f7974198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7973fff640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f797419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 -- 172.21.15.150:0/545681307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974101770 msgr2=0x7f7974198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974101770 0x7f7974198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 -- 172.21.15.150:0/545681307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 msgr2=0x7f797419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:47.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f797419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 -- 172.21.15.150:0/545681307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797419d080 con 0x7f7974102130 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7973fff640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f797419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7972ffd640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7968009230 tx=0x7f796802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968041070 con 0x7f7974102130 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797419d310 con 0x7f7974102130 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7968004450 con 0x7f7974102130 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968003e50 con 0x7f7974102130 2024-03-18T16:12:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.996+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79741a4c00 con 0x7f7974102130 2024-03-18T16:12:47.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.997+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7974068a20 con 0x7f7974102130 2024-03-18T16:12:47.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.998+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) ==== 137422+0+0 (secure 0 0 0) 0x7f7968044070 con 0x7f7974102130 2024-03-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.998+0000 7f7970ff9640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 0x7f795c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.999+0000 7f79737fe640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 0x7f795c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.999+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8187+0+0 (secure 0 0 0) 0x7f79680f0570 con 0x7f7974102130 2024-03-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:46.999+0000 7f79737fe640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 0x7f795c0a4fd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7964009b50 tx=0x7f7964005f90 comp rx=0 tx=0).ready entity=mgr.15336 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:47.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.003+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79680bd530 con 0x7f7974102130 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: Cluster is now healthy 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6836/1835404943,v1:172.21.15.191:6839/1835404943] up:boot 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6837/1604172131,v1:172.21.15.156:6839/1604172131] up:boot 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6832/2311063067,v1:172.21.15.191:6837/2311063067] up:boot 2024-03-18T16:12:47.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.191:6833/1605030190,v1:172.21.15.191:6838/1605030190] up:boot 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6834/92513100,v1:172.21.15.156:6835/92513100] up:boot 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: mds.? [v2:172.21.15.156:6836/3344159386,v1:172.21.15.156:6838/3344159386] up:boot 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: fsmap 12 up:standby 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:12:47.020 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:12:47.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:12:47.021 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:46 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:47.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:12:47.092 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: Cluster is now healthy 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6836/1835404943,v1:172.21.15.191:6839/1835404943] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6837/1604172131,v1:172.21.15.156:6839/1604172131] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6832/2311063067,v1:172.21.15.191:6837/2311063067] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.191:6833/1605030190,v1:172.21.15.191:6838/1605030190] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6834/92513100,v1:172.21.15.156:6835/92513100] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: mds.? [v2:172.21.15.156:6836/3344159386,v1:172.21.15.156:6838/3344159386] up:boot 2024-03-18T16:12:47.093 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: fsmap 12 up:standby 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:12:47.094 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:46 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:47.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.180+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f79741993e0 con 0x7f7974102130 2024-03-18T16:12:47.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:12:47.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:12:47.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: Cluster is now healthy 2024-03-18T16:12:47.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/1685563666' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6836/1835404943,v1:172.21.15.191:6839/1835404943] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6837/1604172131,v1:172.21.15.156:6839/1604172131] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6832/2311063067,v1:172.21.15.191:6837/2311063067] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.191:6833/1605030190,v1:172.21.15.191:6838/1605030190] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6834/92513100,v1:172.21.15.156:6835/92513100] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: mds.? [v2:172.21.15.156:6836/3344159386,v1:172.21.15.156:6838/3344159386] up:boot 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: fsmap 12 up:standby 2024-03-18T16:12:47.214 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='mgr.15336 172.21.15.150:0/4071392818' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:12:47.215 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:46 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:47.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.828+0000 7f7970ff9640 1 -- 172.21.15.150:0/545681307 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7f79680bce30 con 0x7f7974102130 2024-03-18T16:12:47.830 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-03-18T16:12:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.830+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 msgr2=0x7f795c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.830+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 0x7f795c0a4fd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7964009b50 tx=0x7f7964005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.830+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 msgr2=0x7f79741985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.830+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7968009230 tx=0x7f796802cb30 comp rx=0 tx=0).stop 2024-03-18T16:12:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 shutdown_connections 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:6800/669533394,v1:172.21.15.150:6801/669533394] conn(0x7f795c0a2b10 0x7f795c0a4fd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7974107780 0x7f797419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7974102130 0x7f79741985d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 --2- 172.21.15.150:0/545681307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7974101770 0x7f7974198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 >> 172.21.15.150:0/545681307 conn(0x7f7974078be0 msgr2=0x7f79741040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 shutdown_connections 2024-03-18T16:12:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-18T16:12:47.831+0000 7f7979ee8640 1 -- 172.21.15.150:0/545681307 wait complete. 2024-03-18T16:12:47.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:12:47.874 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:12:47.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:12:47.942 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T16:12:47.942 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:12:47.957 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:12:47.974 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:12:47.999 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T16:12:47.999 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T16:12:47.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a 2024-03-18T16:12:48.018 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:47 smithi156 ceph-mon[35919]: pgmap v399: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 11 op/s 2024-03-18T16:12:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:12:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:47 smithi156 ceph-mon[35919]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T16:12:48.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:47 smithi156 ceph-mon[35919]: from='client.? 172.21.15.150:0/545681307' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:48.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:47 smithi150 ceph-mon[31946]: pgmap v399: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 11 op/s 2024-03-18T16:12:48.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:12:48.022 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:47 smithi150 ceph-mon[31946]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T16:12:48.023 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:47 smithi150 ceph-mon[31946]: from='client.? 172.21.15.150:0/545681307' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:48.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:47 smithi191 ceph-mon[36784]: pgmap v399: 97 pgs: 97 active+clean; 594 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 11 op/s 2024-03-18T16:12:48.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/3950286076' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:12:48.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:47 smithi191 ceph-mon[36784]: osdmap e105: 12 total, 12 up, 12 in 2024-03-18T16:12:48.213 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:47 smithi191 ceph-mon[36784]: from='client.? 172.21.15.150:0/545681307' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:12:48.342 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 systemd[1]: Stopping Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:48.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a[31942]: 2024-03-18T16:12:48.391+0000 7f2f06cd2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:12:48.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a[31942]: 2024-03-18T16:12:48.391+0000 7f2f06cd2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T16:12:48.702 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 podman[90262]: 2024-03-18 16:12:48.412769705 +0000 UTC m=+0.158019341 container died 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-18T16:12:49.015 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 podman[90262]: 2024-03-18 16:12:48.808971487 +0000 UTC m=+0.554221131 container cleanup 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:12:49.015 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:48 smithi150 bash[90262]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a 2024-03-18T16:12:49.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T16:12:49.296 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:49 smithi150 podman[90273]: 2024-03-18 16:12:49.010753631 +0000 UTC m=+0.594261027 container remove 71bcd2f54d3176c3a2f8ee167415c2b22d887c2dd04c539a0a877be3f15cc6d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T16:12:49.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:49 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T16:12:49.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:49 smithi150 systemd[1]: Stopped Ceph mon.a for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:12:49.297 INFO:journalctl@ceph.mon.a.smithi150.stdout:Mar 18 16:12:49 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.a.service: Consumed 19.248s CPU time. 2024-03-18T16:12:49.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:49.328 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T16:12:49.328 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T16:12:49.328 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.b 2024-03-18T16:12:49.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:49 smithi156 systemd[1]: Stopping Ceph mon.b for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b[35915]: 2024-03-18T16:12:49.648+0000 7fdd63706700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:12:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b[35915]: 2024-03-18T16:12:49.648+0000 7fdd63706700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T16:12:50.019 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:49 smithi156 podman[72746]: 2024-03-18 16:12:49.782687665 +0000 UTC m=+0.261750366 container died e907da403e8b1b6854077aa739a28a0ed42b383ad72f3fb4a0b7ad77905b1fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:12:50.502 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 podman[72746]: 2024-03-18 16:12:50.307991632 +0000 UTC m=+0.787054321 container cleanup e907da403e8b1b6854077aa739a28a0ed42b383ad72f3fb4a0b7ad77905b1fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:12:50.503 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 bash[72746]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b 2024-03-18T16:12:50.740 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T16:12:50.763 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 podman[72760]: 2024-03-18 16:12:50.501150469 +0000 UTC m=+0.719354718 container remove e907da403e8b1b6854077aa739a28a0ed42b383ad72f3fb4a0b7ad77905b1fab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T16:12:50.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-18T16:12:50.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 systemd[1]: Stopped Ceph mon.b for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:12:50.764 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 18 16:12:50 smithi156 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.b.service: Consumed 11.641s CPU time. 2024-03-18T16:12:50.789 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:50.789 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T16:12:50.789 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T16:12:50.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.c 2024-03-18T16:12:51.114 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:50 smithi191 systemd[1]: Stopping Ceph mon.c for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:51.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-c[36765]: 2024-03-18T16:12:51.112+0000 7fdcf7fa0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:12:51.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-c[36765]: 2024-03-18T16:12:51.112+0000 7fdcf7fa0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T16:12:51.463 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:51 smithi191 podman[74449]: 2024-03-18 16:12:51.317228577 +0000 UTC m=+0.328751227 container died 32f6bc5cc7bb6d374086ba792a2b4ea59eea3d4443cf0327ea4aa83caa35ec62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:12:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:51 smithi191 podman[74449]: 2024-03-18 16:12:51.739296308 +0000 UTC m=+0.750818951 container cleanup 32f6bc5cc7bb6d374086ba792a2b4ea59eea3d4443cf0327ea4aa83caa35ec62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T16:12:51.963 INFO:journalctl@ceph.mon.c.smithi191.stdout:Mar 18 16:12:51 smithi191 bash[74449]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mon-c 2024-03-18T16:12:52.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T16:12:52.178 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:52.179 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T16:12:52.179 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T16:12:52.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x 2024-03-18T16:12:52.490 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:52 smithi150 systemd[1]: Stopping Ceph mgr.x for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:52.842 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:12:52.488+0000 7f92ae553700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:12:52.843 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x[32179]: 2024-03-18T16:12:52.488+0000 7f92ae553700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T16:12:52.843 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:52 smithi150 podman[90384]: 2024-03-18 16:12:52.519476947 +0000 UTC m=+0.149003730 container died 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T16:12:53.342 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 podman[90384]: 2024-03-18 16:12:53.157063077 +0000 UTC m=+0.786589862 container cleanup 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:12:53.343 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 bash[90384]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x 2024-03-18T16:12:53.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T16:12:53.659 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 podman[90398]: 2024-03-18 16:12:53.392380201 +0000 UTC m=+0.871062177 container remove 5a0d9a31a9ca1557973ed5b384ba51095a13c8821266366d3f4f1f5caebe4da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:12:53.659 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-18T16:12:53.659 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 systemd[1]: Stopped Ceph mgr.x for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:12:53.659 INFO:journalctl@ceph.mgr.x.smithi150.stdout:Mar 18 16:12:53 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.x.service: Consumed 1min 19.265s CPU time. 2024-03-18T16:12:53.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:53.673 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T16:12:53.673 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T16:12:53.673 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.y 2024-03-18T16:12:54.001 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:12:53 smithi156 systemd[1]: Stopping Ceph mgr.y for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:54.269 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:12:54 smithi156 podman[72868]: 2024-03-18 16:12:54.000117205 +0000 UTC m=+0.136386956 container died 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:12:54.658 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:12:54 smithi156 podman[72868]: 2024-03-18 16:12:54.413480131 +0000 UTC m=+0.549749862 container cleanup 320f6005c9ae7f3ea94ba661823bed6bd6d13266e274051652e521364dd2f97b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T16:12:54.658 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 18 16:12:54 smithi156 bash[72868]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-y 2024-03-18T16:12:54.775 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T16:12:54.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:54.812 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T16:12:54.812 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T16:12:54.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.z 2024-03-18T16:12:55.150 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:12:54 smithi191 systemd[1]: Stopping Ceph mgr.z for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:55.463 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:12:55 smithi191 podman[74570]: 2024-03-18 16:12:55.148188256 +0000 UTC m=+0.140547210 container died b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:12:55.856 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:12:55 smithi191 podman[74570]: 2024-03-18 16:12:55.569738734 +0000 UTC m=+0.562097678 container cleanup b8274b560dec8d536900b1db1b297bbdf0d2b7d2bacbf047022f4293ff91722f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:12:55.856 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Mar 18 16:12:55 smithi191 bash[74570]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mgr-z 2024-03-18T16:12:55.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T16:12:56.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:12:56.004 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T16:12:56.005 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T16:12:56.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.0 2024-03-18T16:12:56.342 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:12:56 smithi150 systemd[1]: Stopping Ceph osd.0 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:12:56.842 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:12:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0[44221]: 2024-03-18T16:12:56.377+0000 7fcb28068700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:12:56.843 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:12:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0[44221]: 2024-03-18T16:12:56.377+0000 7fcb28068700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-18T16:12:56.843 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:12:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0[44221]: 2024-03-18T16:12:56.377+0000 7fcb28068700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:02.215 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:01 smithi150 podman[90507]: 2024-03-18 16:13:01.901191724 +0000 UTC m=+5.708181382 container died 23424c99f2312cdef83bb9f43f7f449361fe47d6b42be27ae5ad102c32435818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T16:13:02.565 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:02 smithi150 podman[90507]: 2024-03-18 16:13:02.30438534 +0000 UTC m=+6.111375013 container cleanup 23424c99f2312cdef83bb9f43f7f449361fe47d6b42be27ae5ad102c32435818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T16:13:02.565 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:02 smithi150 bash[90507]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0 2024-03-18T16:13:02.843 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:02 smithi150 podman[90523]: 2024-03-18 16:13:02.56414835 +0000 UTC m=+0.661417892 container remove 23424c99f2312cdef83bb9f43f7f449361fe47d6b42be27ae5ad102c32435818 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T16:13:03.153 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:02 smithi150 podman[90618]: 2024-03-18 16:13:02.777567522 +0000 UTC m=+0.017292258 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:03.154 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:02 smithi150 podman[90618]: 2024-03-18 16:13:02.899237883 +0000 UTC m=+0.138962607 container create 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-18T16:13:03.449 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 podman[90618]: 2024-03-18 16:13:03.152156364 +0000 UTC m=+0.391881103 container init 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T16:13:03.450 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 podman[90618]: 2024-03-18 16:13:03.155718235 +0000 UTC m=+0.395442960 container start 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T16:13:03.450 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 podman[90618]: 2024-03-18 16:13:03.260331948 +0000 UTC m=+0.500056680 container attach 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T16:13:03.740 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 podman[90618]: 2024-03-18 16:13:03.448803024 +0000 UTC m=+0.688527749 container died 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:13:03.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T16:13:04.016 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 podman[90643]: 2024-03-18 16:13:03.970465282 +0000 UTC m=+0.512778029 container remove 13c944e5630be45e552425e70624f21ad3a56f7c01ab636559b2d4c2ccd81490 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-18T16:13:04.016 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-18T16:13:04.016 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 systemd[1]: Stopped Ceph osd.0 for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:13:04.016 INFO:journalctl@ceph.osd.0.smithi150.stdout:Mar 18 16:13:03 smithi150 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.0.service: Consumed 1min 3.834s CPU time. 2024-03-18T16:13:04.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:04.033 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T16:13:04.033 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T16:13:04.033 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.1 2024-03-18T16:13:04.344 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:04 smithi156 systemd[1]: Stopping Ceph osd.1 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:04.769 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T16:13:04.341+0000 7fad28099700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:04.769 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T16:13:04.341+0000 7fad28099700 -1 osd.1 106 *** Got signal Terminated *** 2024-03-18T16:13:04.770 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1[40546]: 2024-03-18T16:13:04.341+0000 7fad28099700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:10.145 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:09 smithi156 podman[72993]: 2024-03-18 16:13:09.866318808 +0000 UTC m=+5.644111769 container died 458b9284a3dff382aa88e39897f8186a625eb74d39d37fe96caab0c637cf4d86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-18T16:13:10.519 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:10 smithi156 podman[72993]: 2024-03-18 16:13:10.243194031 +0000 UTC m=+6.020986997 container cleanup 458b9284a3dff382aa88e39897f8186a625eb74d39d37fe96caab0c637cf4d86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T16:13:10.519 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:10 smithi156 bash[72993]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1 2024-03-18T16:13:10.856 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:10 smithi156 podman[73012]: 2024-03-18 16:13:10.519569435 +0000 UTC m=+0.645814749 container remove 458b9284a3dff382aa88e39897f8186a625eb74d39d37fe96caab0c637cf4d86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:13:11.132 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:10 smithi156 podman[73102]: 2024-03-18 16:13:10.757863492 +0000 UTC m=+0.018155523 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:11.133 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:10 smithi156 podman[73102]: 2024-03-18 16:13:10.887224128 +0000 UTC m=+0.147516124 container create 3b19913242b4c51206297c1f79df2094f1c81f7618409c2ddedb89ede9df805b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T16:13:11.436 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:11 smithi156 podman[73102]: 2024-03-18 16:13:11.131595622 +0000 UTC m=+0.391887642 container init 3b19913242b4c51206297c1f79df2094f1c81f7618409c2ddedb89ede9df805b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-18T16:13:11.437 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:11 smithi156 podman[73102]: 2024-03-18 16:13:11.134836696 +0000 UTC m=+0.395128710 container start 3b19913242b4c51206297c1f79df2094f1c81f7618409c2ddedb89ede9df805b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-18T16:13:11.437 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:11 smithi156 podman[73102]: 2024-03-18 16:13:11.223210631 +0000 UTC m=+0.483502648 container attach 3b19913242b4c51206297c1f79df2094f1c81f7618409c2ddedb89ede9df805b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T16:13:11.769 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 18 16:13:11 smithi156 podman[73102]: 2024-03-18 16:13:11.435838449 +0000 UTC m=+0.696130448 container died 3b19913242b4c51206297c1f79df2094f1c81f7618409c2ddedb89ede9df805b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T16:13:12.133 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T16:13:12.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:12.176 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T16:13:12.177 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T16:13:12.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.2 2024-03-18T16:13:12.463 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:12 smithi191 systemd[1]: Stopping Ceph osd.2 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:12.963 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2[41949]: 2024-03-18T16:13:12.492+0000 7f751734a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:12.963 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2[41949]: 2024-03-18T16:13:12.492+0000 7f751734a700 -1 osd.2 106 *** Got signal Terminated *** 2024-03-18T16:13:12.963 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2[41949]: 2024-03-18T16:13:12.492+0000 7f751734a700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:18.151 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:17 smithi191 podman[74694]: 2024-03-18 16:13:17.874381061 +0000 UTC m=+5.476352374 container died 6b00e23ef3735620afe34030f805567db48b3a3ff8653cdca841b5a291f8ed37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T16:13:18.463 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:18 smithi191 podman[74694]: 2024-03-18 16:13:18.248615361 +0000 UTC m=+5.850586657 container cleanup 6b00e23ef3735620afe34030f805567db48b3a3ff8653cdca841b5a291f8ed37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-18T16:13:18.463 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:18 smithi191 bash[74694]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2 2024-03-18T16:13:18.845 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:18 smithi191 podman[74722]: 2024-03-18 16:13:18.550218148 +0000 UTC m=+0.672752969 container remove 6b00e23ef3735620afe34030f805567db48b3a3ff8653cdca841b5a291f8ed37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T16:13:19.172 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:18 smithi191 podman[74814]: 2024-03-18 16:13:18.746863953 +0000 UTC m=+0.017986059 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:19.173 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:18 smithi191 podman[74814]: 2024-03-18 16:13:18.851229999 +0000 UTC m=+0.122352093 container create 162156b5afaf5fd410d8e64a15bcad67975a4cc49c09a700e4e2dc5d735e9a77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-18T16:13:19.173 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:19 smithi191 podman[74814]: 2024-03-18 16:13:19.070685328 +0000 UTC m=+0.341807440 container init 162156b5afaf5fd410d8e64a15bcad67975a4cc49c09a700e4e2dc5d735e9a77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T16:13:19.173 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:19 smithi191 podman[74814]: 2024-03-18 16:13:19.073923179 +0000 UTC m=+0.345045275 container start 162156b5afaf5fd410d8e64a15bcad67975a4cc49c09a700e4e2dc5d735e9a77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T16:13:19.173 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:19 smithi191 podman[74814]: 2024-03-18 16:13:19.170720847 +0000 UTC m=+0.441842944 container attach 162156b5afaf5fd410d8e64a15bcad67975a4cc49c09a700e4e2dc5d735e9a77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T16:13:19.463 INFO:journalctl@ceph.osd.2.smithi191.stdout:Mar 18 16:13:19 smithi191 podman[74814]: 2024-03-18 16:13:19.420350376 +0000 UTC m=+0.691472482 container died 162156b5afaf5fd410d8e64a15bcad67975a4cc49c09a700e4e2dc5d735e9a77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T16:13:20.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T16:13:20.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:20.146 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T16:13:20.146 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T16:13:20.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.3 2024-03-18T16:13:20.475 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:20 smithi150 systemd[1]: Stopping Ceph osd.3 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:20.842 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:20 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3[51166]: 2024-03-18T16:13:20.471+0000 7f8c16579700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:20.842 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:20 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3[51166]: 2024-03-18T16:13:20.471+0000 7f8c16579700 -1 osd.3 106 *** Got signal Terminated *** 2024-03-18T16:13:20.843 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:20 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3[51166]: 2024-03-18T16:13:20.471+0000 7f8c16579700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:22.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:22 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:22.331+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:23.842 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:23 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:23.356+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:24.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:24 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:24.205+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:24.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:24 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:24.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:25.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:25 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:25.161+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:25.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:25.320+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:25.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:25 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:25.172+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:25.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:25.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:26.290 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:26 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:26.159+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:26.291 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:26 smithi150 podman[90734]: 2024-03-18 16:13:26.018807469 +0000 UTC m=+5.672872847 container died 1433e380836b40ee222ceba4caeb686906572c586ded34f25e81d73e054d1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T16:13:26.463 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:25.977+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:26.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:26 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:26.146+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:26.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:26 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:26.288+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:26.593 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:26 smithi150 podman[90734]: 2024-03-18 16:13:26.40675837 +0000 UTC m=+6.060823759 container cleanup 1433e380836b40ee222ceba4caeb686906572c586ded34f25e81d73e054d1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T16:13:26.593 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:26 smithi150 bash[90734]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3 2024-03-18T16:13:26.951 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:26.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:27.019 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:26 smithi150 podman[90751]: 2024-03-18 16:13:26.675220625 +0000 UTC m=+0.650681145 container remove 1433e380836b40ee222ceba4caeb686906572c586ded34f25e81d73e054d1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:13:27.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:26.949+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:27.304 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:27.161+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:27.305 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:26.921983381 +0000 UTC m=+0.018404668 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:27.305 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:27.042771013 +0000 UTC m=+0.139192280 container create 0385a7b4bbd92019dbffb7e4b5cf5c8dc8574740e589c92bb2b2344bc28ac138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T16:13:27.305 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:27.303849152 +0000 UTC m=+0.400270435 container init 0385a7b4bbd92019dbffb7e4b5cf5c8dc8574740e589c92bb2b2344bc28ac138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T16:13:27.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:27 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:27.137+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:27.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:27 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:27.310+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:27.593 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:27.307177521 +0000 UTC m=+0.403598799 container start 0385a7b4bbd92019dbffb7e4b5cf5c8dc8574740e589c92bb2b2344bc28ac138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T16:13:27.593 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:27.395608826 +0000 UTC m=+0.492030094 container attach 0385a7b4bbd92019dbffb7e4b5cf5c8dc8574740e589c92bb2b2344bc28ac138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T16:13:27.909 INFO:journalctl@ceph.osd.3.smithi150.stdout:Mar 18 16:13:27 smithi150 podman[90843]: 2024-03-18 16:13:27.594093785 +0000 UTC m=+0.690515054 container died 0385a7b4bbd92019dbffb7e4b5cf5c8dc8574740e589c92bb2b2344bc28ac138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-18T16:13:27.911 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:27.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:28.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:27.909+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:28.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T16:13:28.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:28.287 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T16:13:28.287 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T16:13:28.288 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.4 2024-03-18T16:13:28.497 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:28 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:28.166+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:28.498 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:28 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:28.290+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:28.519 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:28 smithi156 systemd[1]: Stopping Ceph osd.4 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:28.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:28 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:28.166+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:28.921 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:28.558+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:29.019 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:28 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4[47369]: 2024-03-18T16:13:28.588+0000 7efec2f77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:29.019 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:28 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4[47369]: 2024-03-18T16:13:28.588+0000 7efec2f77700 -1 osd.4 106 *** Got signal Terminated *** 2024-03-18T16:13:29.019 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:28 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4[47369]: 2024-03-18T16:13:28.588+0000 7efec2f77700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:29.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:28.919+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:29.422 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:29.119+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:29.423 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:29 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:29.278+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:29.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:29 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:29.134+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:29.881 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:29 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:29.606+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:30.150 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:29 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:29.879+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:30.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:30 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:30.096+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:30.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:30 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:30.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:30.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:30 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:30.119+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:30.848 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:30 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:30.584+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:31.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:30 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:30.846+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:31.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:31 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:31.048+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:31.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:31 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:31.153+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:31.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:31 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:31.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:31.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:31 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:31.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:31.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:31 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:31.806+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:31.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:31 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:31.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:31.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:31 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:31.825+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:32.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:32 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:32.080+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:32.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:32 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:32.162+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:32.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:32 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:32.166+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:32.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:32 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:32.167+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:32.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:32 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:32.799+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:32.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:32 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:32.798+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:32.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:32 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:32.570+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:33.494 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:33 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:33.206+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:33.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:33 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:33.128+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:33.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:33 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:33.128+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:33.518 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:33 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:33.118+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:33.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:33 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:33.802+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:33.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:33 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:33.565+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:33.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:33 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:33.758+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:34.315 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:34 smithi156 podman[73234]: 2024-03-18 16:13:34.031117955 +0000 UTC m=+5.552443729 container died 2326ab523bc75c0027b3039c07b7e1f0f5ca7ebb82f08e04df07cef535b7cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T16:13:34.315 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:34 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:34.070+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:34.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:34.204+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:34.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:34.204+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:34.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:34.099+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:34.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:34 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:34.099+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:34.637 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:34 smithi156 podman[73234]: 2024-03-18 16:13:34.405048364 +0000 UTC m=+5.926374143 container cleanup 2326ab523bc75c0027b3039c07b7e1f0f5ca7ebb82f08e04df07cef535b7cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T16:13:34.638 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:34 smithi156 bash[73234]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4 2024-03-18T16:13:34.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:34 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:34.573+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:34.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:34 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:34.573+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:34.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:34 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:34.791+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:34.964 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:34 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:34.791+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:34.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:34 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:34.740+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:35.019 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:34 smithi156 podman[73251]: 2024-03-18 16:13:34.72341434 +0000 UTC m=+0.685834616 container remove 2326ab523bc75c0027b3039c07b7e1f0f5ca7ebb82f08e04df07cef535b7cfd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:13:35.337 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:34.965244774 +0000 UTC m=+0.017769618 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:35.338 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:35.099427052 +0000 UTC m=+0.151951876 container create 7e8ab99f8394259c0f8dc4db00990036884613971ceea5ccb53425b8b402e55f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T16:13:35.338 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:35.335241589 +0000 UTC m=+0.387766413 container init 7e8ab99f8394259c0f8dc4db00990036884613971ceea5ccb53425b8b402e55f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T16:13:35.339 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:35 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:35.116+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:35.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:35 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:35.157+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:35.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:35 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:35.157+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:35.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:35 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:35.106+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:35.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:35 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:35.106+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:35.631 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:35.338452505 +0000 UTC m=+0.390977341 container start 7e8ab99f8394259c0f8dc4db00990036884613971ceea5ccb53425b8b402e55f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T16:13:35.631 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:35.451873905 +0000 UTC m=+0.504398747 container attach 7e8ab99f8394259c0f8dc4db00990036884613971ceea5ccb53425b8b402e55f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T16:13:35.932 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 18 16:13:35 smithi156 podman[73341]: 2024-03-18 16:13:35.630428346 +0000 UTC m=+0.682953176 container died 7e8ab99f8394259c0f8dc4db00990036884613971ceea5ccb53425b8b402e55f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-18T16:13:35.932 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:35 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:35.712+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:35.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:35 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:35.716+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:35.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:35 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:35.823+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:35.964 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:35 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:35.823+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:35.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:35 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:35.583+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:35.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:35 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:35.583+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:36.231 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:36 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:36.094+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:36.232 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:36 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:36.094+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:36.254 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T16:13:36.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:36.295 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T16:13:36.295 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T16:13:36.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.5 2024-03-18T16:13:36.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:36 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:36.197+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:36.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:36 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:36.197+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:36.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:36 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:36.155+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:36.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:36 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:36.155+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:36.612 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 systemd[1]: Stopping Ceph osd.5 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:36.612 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:36.549+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:36.613 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:36.549+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:36.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:36.752+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:36.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:36.610+0000 7f31e4dd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:36.963 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:36.610+0000 7f31e4dd6700 -1 osd.5 106 *** Got signal Terminated *** 2024-03-18T16:13:36.964 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:36.610+0000 7f31e4dd6700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:36.964 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:36.809+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:36.964 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:36 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:36.809+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:37.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:36 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:36.720+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:37.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:37 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:37.058+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:37.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:37 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:37.058+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:37.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:37.186+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:37.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:37.186+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:37.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:37.226+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:37.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:37.226+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:37.824 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:37 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:37.734+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:37.824 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:37 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:37.512+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:37.824 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:37 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:37.512+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:38.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:37 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:37.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:38.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:37 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:37.821+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:38.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:37 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:37.821+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:38.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:38.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:38.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:38.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:38.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:38.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:38.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:38 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:38.179+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:38.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:38 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:38.179+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:38.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:38 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:38.148+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:38.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:38 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:38.148+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:38.824 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:38 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:38.548+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:38.825 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:38 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:38.548+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:38.825 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:38 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:38.699+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:39.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:38 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:38.711+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:39.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:38 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:38.822+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:39.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:38 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:38.822+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:39.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:39 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:39.085+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:39.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:39 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:39.085+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:39.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:39 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:39.085+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:39.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:39 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:39.179+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:39.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:39 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:39.179+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:39.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:39 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:39.182+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:39.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:39 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:39.182+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:39.830 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:39.524+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:39.830 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:39.524+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:39.830 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:39.695+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:40.018 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:39 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:39.672+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:40.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:39.828+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:40.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:39.828+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:40.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:39 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:39.828+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.042141+0000 front 2024-03-18T16:13:15.042174+0000 (oldest deadline 2024-03-18T16:13:39.742347+0000) 2024-03-18T16:13:40.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:40 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:40.045+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:40.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:40 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:40.045+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:40.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:40 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:40.045+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:40.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:40.163+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:40.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:40.163+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:40.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:40.167+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:40.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:40.167+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:40.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:40.167+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:40.867 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:40.686+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:40.867 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:40.476+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:40.867 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:40.476+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:40.868 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:40.476+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:41.004 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:40 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:40.666+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:41.004 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:40 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:40.666+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:41.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:40.865+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.638221+0000 front 2024-03-18T16:13:00.638163+0000 (oldest deadline 2024-03-18T16:13:25.838993+0000) 2024-03-18T16:13:41.213 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:40.865+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.640997+0000 front 2024-03-18T16:13:08.641024+0000 (oldest deadline 2024-03-18T16:13:34.541130+0000) 2024-03-18T16:13:41.214 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:40 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5[48918]: 2024-03-18T16:13:40.865+0000 7f31dd3c2700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.042141+0000 front 2024-03-18T16:13:15.042174+0000 (oldest deadline 2024-03-18T16:13:39.742347+0000) 2024-03-18T16:13:41.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.002+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:41.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.002+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:41.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.002+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:41.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:41.177+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:41.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:41.177+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:41.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:41.192+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:41.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:41.192+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:41.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:41 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:41.192+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:41.709 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:41 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:41.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:41.709 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:41 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:41.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:41.709 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:41 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:41.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:41.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:41 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:41.707+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:41.998 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:41.701+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:41.998 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:41.701+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:42.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.996+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:42.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.996+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:42.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:41 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:41.996+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:42.422 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:42 smithi191 podman[74919]: 2024-03-18 16:13:42.087371234 +0000 UTC m=+5.597448179 container died d807845a890fad327f323271361f961c05d2a493f0e48d2a244bb42ebfc159b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T16:13:42.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:42 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:42.162+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:42.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:42 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:42.162+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:42.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:42 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:42.151+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:42.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:42 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:42.151+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:42.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:42 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:42.151+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:42.713 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:42 smithi191 podman[74919]: 2024-03-18 16:13:42.519069076 +0000 UTC m=+6.029146031 container cleanup d807845a890fad327f323271361f961c05d2a493f0e48d2a244bb42ebfc159b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T16:13:42.713 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:42 smithi191 bash[74919]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5 2024-03-18T16:13:42.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:42 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:42.478+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:42.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:42 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:42.478+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:42.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:42 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:42.478+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:43.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:42 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:42.704+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:43.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:42 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:42.704+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:43.169 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:42 smithi191 podman[74936]: 2024-03-18 16:13:42.7617938 +0000 UTC m=+0.669997525 container remove d807845a890fad327f323271361f961c05d2a493f0e48d2a244bb42ebfc159b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T16:13:43.169 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:42 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:42.717+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:43.457 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.070978325 +0000 UTC m=+0.017959655 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:43.458 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.179309703 +0000 UTC m=+0.126291016 container create dbaac24316beab6022f7b5950687652108afc12215fc056ad6b50815aba9b4e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T16:13:43.458 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.382077313 +0000 UTC m=+0.329058628 container init dbaac24316beab6022f7b5950687652108afc12215fc056ad6b50815aba9b4e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T16:13:43.458 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.38534535 +0000 UTC m=+0.332326655 container start dbaac24316beab6022f7b5950687652108afc12215fc056ad6b50815aba9b4e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:13:43.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:43 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:43.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:43.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:43 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:43.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:43.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:43 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:43.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:43.545 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:43 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:43.185+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:43.546 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:43 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:43.185+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:43.546 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:43 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:43.150+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:43.546 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:43 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:43.150+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:43.546 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:43 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:43.150+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:43.713 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.485417862 +0000 UTC m=+0.432399185 container attach dbaac24316beab6022f7b5950687652108afc12215fc056ad6b50815aba9b4e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T16:13:43.714 INFO:journalctl@ceph.osd.5.smithi191.stdout:Mar 18 16:13:43 smithi191 podman[75030]: 2024-03-18 16:13:43.679902143 +0000 UTC m=+0.626883473 container died dbaac24316beab6022f7b5950687652108afc12215fc056ad6b50815aba9b4e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T16:13:43.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:43 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:43.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:43.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:43 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:43.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:43.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:43 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:43.455+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:44.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:43 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:43.669+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:44.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:43 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:43.669+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:44.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:43 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:43.753+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:44.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:43 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:43.753+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:44.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T16:13:44.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:44.377 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T16:13:44.377 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T16:13:44.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.6 2024-03-18T16:13:44.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:44.048+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:44.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:44.048+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:44.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:44.048+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:44.586 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:44.487+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:44.587 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:44.487+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:44.587 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:44.487+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:44.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:44.187+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:44.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:44.187+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:44.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:44.187+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:44.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.219+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:44.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.219+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:44.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.219+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:44.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 systemd[1]: Stopping Ceph osd.6 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:44.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:44.755+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:44.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:44 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:44.755+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:45.018 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:44.630+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:45.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:44 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:44.630+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:45.092 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.711+0000 7efe2b8dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:45.092 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.711+0000 7efe2b8dc700 -1 osd.6 106 *** Got signal Terminated *** 2024-03-18T16:13:45.093 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:44 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:44.711+0000 7efe2b8dc700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:45.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:45 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:45.078+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:45.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:45 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:45.078+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:45.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:45 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:45.078+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:45.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:45.205+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:45.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:45.205+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:45.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:45.205+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:45.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:45.213+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:45.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:45.213+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:45.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:45 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:45.213+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:45.753 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:45 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:45.497+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:45.754 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:45 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:45.497+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:45.754 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:45 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:45.497+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:46.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:45 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:45.659+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:46.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:45 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:45.659+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:46.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:45 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:45.751+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:46.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:45 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:45.751+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:46.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:46 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:46.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:46.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:46 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:46.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:46.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:46 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:46.099+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:46.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:46.198+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:46.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:46.198+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:46.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:46.198+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:46.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:46.189+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:46.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:46.189+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:46.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:46.189+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:46.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:46 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:46.728+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:46.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:46 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:46.728+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:46.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:46 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:46.536+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:46.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:46 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:46.536+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:46.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:46 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:46.536+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:47.018 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:46 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:46.698+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:47.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:46 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:46.698+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:47.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:47.123+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:47.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:47.123+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:47.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:47.123+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:47.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:47.183+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:47.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:47.183+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:47.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:47.183+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:47.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:47.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:47.594 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:47.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:47.594 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:47.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:47.594 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:47 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:47.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:47.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:47.726+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:47.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:47.726+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:47.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:47.539+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:47.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:47.539+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:47.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:47 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:47.539+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:48.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:47.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:48.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:47.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:48.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:47 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:47.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:48.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:48.159+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:48.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:48.159+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:48.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:48.159+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:48.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:48.159+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:48.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:48.158+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:48.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:48.158+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:48.158+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:48.158+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.289626+0000 front 2024-03-18T16:13:23.289643+0000 (oldest deadline 2024-03-18T16:13:47.989808+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:48.221+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:48.221+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:48.221+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:48.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:48 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:48.221+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:48.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:48.531+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:48.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:48.531+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:48.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:48.531+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:48.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:48.698+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:48.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:48.698+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:48.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:48 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:48.698+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:49.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:48.694+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:49.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:48.694+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:49.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:48 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:48.694+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:49.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:49.183+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:49.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:49.183+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:49.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:49.183+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:49.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:49.183+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:49.592 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:49.178+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.086297+0000 front 2024-03-18T16:12:58.086281+0000 (oldest deadline 2024-03-18T16:13:25.087162+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:49.178+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.588301+0000 front 2024-03-18T16:13:08.588243+0000 (oldest deadline 2024-03-18T16:13:33.888202+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:49.178+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:17.389364+0000 front 2024-03-18T16:13:17.389250+0000 (oldest deadline 2024-03-18T16:13:43.289233+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6[57722]: 2024-03-18T16:13:49.178+0000 7efe27ed0700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.289626+0000 front 2024-03-18T16:13:23.289643+0000 (oldest deadline 2024-03-18T16:13:47.989808+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:49.207+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:49.207+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:49.207+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:49.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:49 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:49.207+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:49.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:49.692+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:49.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:49.692+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:49.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:49.692+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:49.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:49.534+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:49.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:49.534+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:49.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:49 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:49.534+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:50.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:49.663+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:50.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:49.663+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:50.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:49 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:49.663+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:50.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:50.228+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:50.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:50.228+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:50.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:50.228+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:50.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:50.229+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:50.581 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:50 smithi150 podman[90960]: 2024-03-18 16:13:50.288050391 +0000 UTC m=+5.716460469 container died 7a7e84cad1d3c23808743827a8ed2167ca1bcf446121cedd4d38ef2e63df0843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T16:13:50.581 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:50.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:50.582 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:50.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:50.582 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:50.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:50.582 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:50 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:50.214+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:50.839 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:50 smithi150 podman[90960]: 2024-03-18 16:13:50.670150481 +0000 UTC m=+6.098560558 container cleanup 7a7e84cad1d3c23808743827a8ed2167ca1bcf446121cedd4d38ef2e63df0843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T16:13:50.839 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:50 smithi150 bash[90960]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6 2024-03-18T16:13:50.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:50.568+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:50.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:50.568+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:50.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:50.568+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:50.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:50.670+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:50.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:50.670+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:50.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:50 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:50.670+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:51.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:50.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:51.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:50.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:51.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:50.700+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:51.092 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:50 smithi150 podman[90977]: 2024-03-18 16:13:50.838095112 +0000 UTC m=+0.554137231 container remove 7a7e84cad1d3c23808743827a8ed2167ca1bcf446121cedd4d38ef2e63df0843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T16:13:51.459 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.130615751 +0000 UTC m=+0.017841829 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:51.459 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.239235191 +0000 UTC m=+0.126461307 container create c351babcb58a439f2c1f998ed4e77afb957077e9fadb41562226458ae4a9e7d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T16:13:51.459 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.458339273 +0000 UTC m=+0.345565354 container init c351babcb58a439f2c1f998ed4e77afb957077e9fadb41562226458ae4a9e7d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-18T16:13:51.460 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:51.206+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:51.460 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:51.206+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:51.460 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:51.206+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:51.460 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:51 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:51.206+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:51.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:51.259+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:51.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:51.259+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:51.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:51.259+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:51.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:51.259+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:51.753 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.461483693 +0000 UTC m=+0.348709765 container start c351babcb58a439f2c1f998ed4e77afb957077e9fadb41562226458ae4a9e7d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T16:13:51.753 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.541584334 +0000 UTC m=+0.428810427 container attach c351babcb58a439f2c1f998ed4e77afb957077e9fadb41562226458ae4a9e7d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:13:51.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:51.546+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:51.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:51.546+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:51.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:51.546+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:51.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:51.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:51.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:51.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:51.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:51 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:51.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:52.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:51.670+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:52.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:51.670+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:52.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:51 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:51.670+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:52.055 INFO:journalctl@ceph.osd.6.smithi150.stdout:Mar 18 16:13:51 smithi150 podman[91072]: 2024-03-18 16:13:51.752719701 +0000 UTC m=+0.639945783 container died c351babcb58a439f2c1f998ed4e77afb957077e9fadb41562226458ae4a9e7d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:13:52.342 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:52.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:52.343 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:52.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:52.343 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:52.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:52.343 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:52 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:52.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:52.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T16:13:52.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:13:52.441 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T16:13:52.442 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T16:13:52.442 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.7 2024-03-18T16:13:52.471 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:52.216+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:52.471 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:52.216+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:52.471 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:52.216+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:52.471 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:52.216+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:52.763 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 systemd[1]: Stopping Ceph osd.7 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:13:52.764 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.644+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:52.764 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.644+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:52.764 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.644+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:52.764 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.761+0000 7fa1db4da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:13:52.764 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.761+0000 7fa1db4da700 -1 osd.7 106 *** Got signal Terminated *** 2024-03-18T16:13:52.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:52.559+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:52.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:52.559+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:52.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:52.559+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:52.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:52.661+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:52.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:52.661+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:52.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:52 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:52.661+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:53.018 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:52 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:52.761+0000 7fa1db4da700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:13:53.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:53.214+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:53.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:53.214+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:53.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:53.214+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:53.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:53.214+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:53.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:53.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:53.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:53.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:53.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:53.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:53.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:53 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:53.191+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:53.793 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:53.610+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:53.793 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:53.610+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:53.793 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:53 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:53.610+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:53.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:53.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:53.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:53.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:53.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:53.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:53.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:53.623+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:53.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:53.623+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:53.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:53 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:53.623+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:54.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:54.165+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:54.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:54.165+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:54.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:54.165+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:54.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:54.165+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:54.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:54 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:54.235+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:54.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:54 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:54.235+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:54.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:54 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:54.235+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:54.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:54 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:54.235+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:54.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:54.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:54.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:54.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:54.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:54.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:54.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:54.629+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:54.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:54.629+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:54.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:54 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:54.629+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:55.018 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:54.640+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:55.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:54.640+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:55.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:54.640+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:55.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:55.151+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:55.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:55.151+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:55.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:55.151+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:55.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:55.151+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:55.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:55.259+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:55.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:55.259+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:55.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:55.259+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:55.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:55 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:55.259+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:55.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:55.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:55.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:55.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:55.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:55.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:55.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:55.665+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:55.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:55.665+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:55.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:55 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:55.665+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:56.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:55.596+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:56.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:55.596+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:56.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:55 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:55.596+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:56.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:56.130+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:56.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:56.130+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:56.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:56.130+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:56.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:56.130+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:56.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:56.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:56.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:56.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:56.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:56.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:56.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:56 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:56.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:56.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:56.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:56.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:56.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:56.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:56.637+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:56.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:56.597+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:56.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:56.597+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:56.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:56.597+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:56.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:56 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:56.597+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:13:57.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:56.620+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:57.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:56.620+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:57.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:56 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:56.620+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:57.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:57.095+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:57.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:57.095+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:57.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:57.095+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:57.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:57.095+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:57.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:57.095+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:13:57.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:57.254+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:57.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:57.254+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:57.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:57.254+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:57.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:57 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:57.254+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:57.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:57.596+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:57.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:57.596+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:57.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:57.596+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:57.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:57.596+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:13:57.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:57.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:57.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:57.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:57.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:57.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:57.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:57 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:57.582+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:13:58.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:57.642+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:07.804349+0000 front 2024-03-18T16:13:07.804503+0000 (oldest deadline 2024-03-18T16:13:35.405245+0000) 2024-03-18T16:13:58.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:57.642+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.405891+0000 front 2024-03-18T16:13:15.405916+0000 (oldest deadline 2024-03-18T16:13:40.106106+0000) 2024-03-18T16:13:58.019 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:57 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7[54333]: 2024-03-18T16:13:57.642+0000 7fa1d3ac6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:21.207155+0000 front 2024-03-18T16:13:21.207185+0000 (oldest deadline 2024-03-18T16:13:47.107214+0000) 2024-03-18T16:13:58.519 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:58 smithi156 podman[73446]: 2024-03-18 16:13:58.276760891 +0000 UTC m=+5.644528216 container died 8196aefa866913213f23d8634de01c4c6ca86540b5d01da8819863b07b5935d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:13:58.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:58.079+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:58.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:58.079+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:58.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:58.079+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:58.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:58.079+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:58.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:58.079+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:13:58.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:58.213+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:58.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:58.213+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:58.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:58.213+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:58.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:58 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:58.213+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:58.885 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:58 smithi156 podman[73446]: 2024-03-18 16:13:58.666554552 +0000 UTC m=+6.034321875 container cleanup 8196aefa866913213f23d8634de01c4c6ca86540b5d01da8819863b07b5935d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:13:58.885 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:58 smithi156 bash[73446]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7 2024-03-18T16:13:58.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:58.605+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:58.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:58.605+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:58.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:58.605+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:58.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:58.605+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:13:58.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:58.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:58.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:58.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:58.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:58.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:58.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:58 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:58.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:13:59.208 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:58 smithi156 podman[73475]: 2024-03-18 16:13:58.884283564 +0000 UTC m=+0.608516805 container remove 8196aefa866913213f23d8634de01c4c6ca86540b5d01da8819863b07b5935d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T16:13:59.209 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:59.081+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:13:59.209 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:59.081+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:13:59.209 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:59.081+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:13:59.209 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:59.081+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:13:59.209 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:13:59 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:13:59.081+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:13:59.514 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.110488661 +0000 UTC m=+0.018585658 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:13:59.514 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.26019095 +0000 UTC m=+0.168287933 container create e0e3e3ee524ff4479ca01276b15308d959e37f3a0ef20b63fe28a137e30dff58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:13:59.514 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.51300887 +0000 UTC m=+0.421105854 container init e0e3e3ee524ff4479ca01276b15308d959e37f3a0ef20b63fe28a137e30dff58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True) 2024-03-18T16:13:59.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:59.190+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:13:59.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:59.190+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:13:59.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:59.190+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:13:59.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:13:59 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:13:59.190+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:13:59.769 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.516535072 +0000 UTC m=+0.424632061 container start e0e3e3ee524ff4479ca01276b15308d959e37f3a0ef20b63fe28a137e30dff58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T16:13:59.769 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.596254563 +0000 UTC m=+0.504351557 container attach e0e3e3ee524ff4479ca01276b15308d959e37f3a0ef20b63fe28a137e30dff58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T16:13:59.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:59.560+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:13:59.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:59.560+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:59.560+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:13:59.560+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:59.567+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:59.567+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:59.567+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:13:59.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:13:59 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:13:59.567+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:00.092 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 18 16:13:59 smithi156 podman[73563]: 2024-03-18 16:13:59.80469005 +0000 UTC m=+0.712787042 container died e0e3e3ee524ff4479ca01276b15308d959e37f3a0ef20b63fe28a137e30dff58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T16:14:00.092 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:00.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:00.092 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:00.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:00.092 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:00.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:00.093 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:00.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:00.093 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:00 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:00.040+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:00.395 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T16:14:00.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:00.478 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T16:14:00.478 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T16:14:00.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.8 2024-03-18T16:14:00.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:00.196+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:00.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:00.196+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:00.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:00.196+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:00.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:00 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:00.196+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:00.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:00.580+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:00.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:00.580+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:00.798 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:00.580+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:00.580+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 systemd[1]: Stopping Ceph osd.8 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.574+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.574+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.574+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:00.799 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.574+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:00.800 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.796+0000 7fa31e5ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:00.800 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.796+0000 7fa31e5ff700 -1 osd.8 106 *** Got signal Terminated *** 2024-03-18T16:14:00.800 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:00 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:00.796+0000 7fa31e5ff700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:14:01.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.022+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:01.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.022+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:01.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.022+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:01.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.022+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:01.520 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.022+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:01.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:01.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:01.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:01.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:01.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:01.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:01.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:01 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:01.239+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:01.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:01.608+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:01.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:01.608+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:01.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:01.608+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:01.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:01.608+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:01.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:01.606+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:01.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:01.606+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:01.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:01.606+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:01.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:01 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:01.606+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:02.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.995+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:02.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.995+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:02.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.995+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:02.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.995+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:02.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:01.995+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:02.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:02.209+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:02.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:02.209+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:02.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:02.209+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:02.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:02 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:02.209+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:02.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:02.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:02.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:02.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:02.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:02.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:02.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:02.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.410190+0000 front 2024-03-18T16:13:41.410158+0000 (oldest deadline 2024-03-18T16:14:02.510339+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:02.612+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:02.612+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:02.612+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:02.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:02 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:02.612+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:03.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:03.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:03.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:03.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:03.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:03.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:02 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:02.976+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:03.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:03 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:03.216+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:03.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:03 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:03.216+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:03.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:03 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:03.216+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:03.494 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:03 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:03.216+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:03.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:03.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:03.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:03.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:03.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:03.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:03.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:03.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:03.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:03.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:03.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:03.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:03.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:03.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:03.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:03.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:03.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:03 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:03.586+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.410190+0000 front 2024-03-18T16:13:41.410158+0000 (oldest deadline 2024-03-18T16:14:02.510339+0000) 2024-03-18T16:14:04.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:04.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:04.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:04.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:04.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:04.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:03 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:03.960+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:04.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:04.238+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:04.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:04.238+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:04.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:04.238+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:04.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:04 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:04.238+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:04.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:04.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:04.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:04.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:04.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:04.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:04.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:04.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:04.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:04.558+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:04.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:04.558+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:04.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:04.558+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:04.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:04.558+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:04.965 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:04 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:04.558+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.410190+0000 front 2024-03-18T16:13:41.410158+0000 (oldest deadline 2024-03-18T16:14:02.510339+0000) 2024-03-18T16:14:05.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:05.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:05.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:05.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:05.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:05.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:04 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:04.966+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:05.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:05 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:05.233+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:05.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:05 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:05.233+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:05.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:05 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:05.233+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:05.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:05 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:05.233+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:05.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:05.577+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.303408+0000 front 2024-03-18T16:13:08.303621+0000 (oldest deadline 2024-03-18T16:13:31.203433+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:05.577+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.906081+0000 front 2024-03-18T16:13:16.906108+0000 (oldest deadline 2024-03-18T16:13:42.806248+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:05.577+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:23.307484+0000 front 2024-03-18T16:13:23.307598+0000 (oldest deadline 2024-03-18T16:13:48.508170+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:05.577+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:28.508894+0000 front 2024-03-18T16:13:28.508855+0000 (oldest deadline 2024-03-18T16:13:57.309201+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8[55704]: 2024-03-18T16:14:05.577+0000 7fa316beb700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.410190+0000 front 2024-03-18T16:13:41.410158+0000 (oldest deadline 2024-03-18T16:14:02.510339+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:05.579+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:05.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:05.579+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:05.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:05.579+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:05.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:05.579+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:05.965 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:05 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:05.579+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:06.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:06.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:06.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:06.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:06.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:06.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:05 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:05.932+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:06.540 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:06 smithi191 podman[75147]: 2024-03-18 16:14:06.272504856 +0000 UTC m=+5.598142109 container died c25666d668d49393a35cf6f446482b136fc2271c3bb5c29af58c66d1f392f3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T16:14:06.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:06.228+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:06.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:06.228+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:06.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:06.228+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:06.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:06 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:06.228+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:06.807 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:06 smithi191 podman[75147]: 2024-03-18 16:14:06.604339868 +0000 UTC m=+5.929977120 container cleanup c25666d668d49393a35cf6f446482b136fc2271c3bb5c29af58c66d1f392f3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T16:14:06.807 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:06 smithi191 bash[75147]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8 2024-03-18T16:14:06.808 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:06.614+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:06.808 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:06.614+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:06.808 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:06.614+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:06.808 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:06.614+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:06.808 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:06 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:06.614+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:07.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:06 smithi191 podman[75164]: 2024-03-18 16:14:06.805449714 +0000 UTC m=+0.532160699 container remove c25666d668d49393a35cf6f446482b136fc2271c3bb5c29af58c66d1f392f3a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:07.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:06 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:06.922+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:07.552 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.120555382 +0000 UTC m=+0.017556833 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:14:07.552 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.239930768 +0000 UTC m=+0.136932218 container create ac8fb9eb75356ff0f40ac3fc99863d2df2a26d1cf6a685041910244f606cd28e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T16:14:07.553 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.458764972 +0000 UTC m=+0.355766422 container init ac8fb9eb75356ff0f40ac3fc99863d2df2a26d1cf6a685041910244f606cd28e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:14:07.553 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.461981833 +0000 UTC m=+0.358983281 container start ac8fb9eb75356ff0f40ac3fc99863d2df2a26d1cf6a685041910244f606cd28e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:14:07.553 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.550417679 +0000 UTC m=+0.447419138 container attach ac8fb9eb75356ff0f40ac3fc99863d2df2a26d1cf6a685041910244f606cd28e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:14:07.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:07.268+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:07.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:07.268+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:07.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:07.268+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:07.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:07.268+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:07.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:07 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:07.268+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:07.963 INFO:journalctl@ceph.osd.8.smithi191.stdout:Mar 18 16:14:07 smithi191 podman[75259]: 2024-03-18 16:14:07.751739581 +0000 UTC m=+0.648741033 container died ac8fb9eb75356ff0f40ac3fc99863d2df2a26d1cf6a685041910244f606cd28e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-18T16:14:07.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:07.627+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:07.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:07.627+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:07.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:07.627+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:07.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:07.627+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:07.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:07.627+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:08.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:07 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:07.896+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:08.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T16:14:08.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:08.473 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T16:14:08.474 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T16:14:08.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.9 2024-03-18T16:14:08.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:08.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:08.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:08.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:08.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.306+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:08.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 systemd[1]: Stopping Ceph osd.9 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:08.681 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:08.637+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:08.681 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:08.637+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:08.681 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:08.637+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:08.681 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:08.637+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:08.681 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:08 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:08.637+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:09.092 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.780+0000 7faa0ede8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:09.092 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.780+0000 7faa0ede8700 -1 osd.9 106 *** Got signal Terminated *** 2024-03-18T16:14:09.093 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:08 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:08.780+0000 7faa0ede8700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:14:09.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:09.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:09.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:09.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:09.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:09.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:08 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:08.906+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:09.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:09.300+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:09.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:09.300+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:09.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:09.300+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:09.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:09.300+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:09.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:09 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:09.300+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:09.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:09.635+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:09.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:09.635+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:09.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:09.635+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:09.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:09.635+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:09.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:09 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:09.635+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:10.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:10.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:10.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:10.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:10.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:10.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:09 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:09.921+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:10.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:10.253+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:10.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:10.253+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:10.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:10.253+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:10.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:10.253+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:10.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:10 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:10.253+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:10.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:10.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:10.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:10.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:10.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:10.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:10.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:10.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:10.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:10.616+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:11.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:11.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:11.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:11.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:11.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:11.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:10 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:10.900+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:11.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:11.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:11.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:11.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:11.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:11.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:11.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:11.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:11.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:11 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:11.237+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:11.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:11 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:11.602+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:11.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:11 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:11.602+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:11.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:11 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:11.602+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:11.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:11 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:11.602+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:11.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:11 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:11.602+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:12.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:12.271 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:12.271 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:12.271 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:12.272 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:12.272 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:12.272 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:11 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:11.898+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:12.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:12.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:12.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:12.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:12.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:12.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:12 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:12.232+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.556537+0000 front 2024-03-18T16:13:46.556598+0000 (oldest deadline 2024-03-18T16:14:11.856607+0000) 2024-03-18T16:14:12.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:12.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:12.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:12.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:12.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:12.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:12.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:12.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:12.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:12 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:12.641+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:13.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:13.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:12 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:12.861+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:13.592 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:59.646185+0000 front 2024-03-18T16:12:59.646217+0000 (oldest deadline 2024-03-18T16:13:21.946382+0000) 2024-03-18T16:14:13.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:05.848417+0000 front 2024-03-18T16:13:05.848510+0000 (oldest deadline 2024-03-18T16:13:30.548514+0000) 2024-03-18T16:14:13.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.150730+0000 front 2024-03-18T16:13:15.150742+0000 (oldest deadline 2024-03-18T16:13:39.850743+0000) 2024-03-18T16:14:13.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:22.152239+0000 front 2024-03-18T16:13:22.152376+0000 (oldest deadline 2024-03-18T16:13:46.252397+0000) 2024-03-18T16:14:13.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:41.256256+0000 front 2024-03-18T16:13:41.256221+0000 (oldest deadline 2024-03-18T16:14:06.556183+0000) 2024-03-18T16:14:13.593 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:13 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9[64770]: 2024-03-18T16:14:13.272+0000 7faa073d4700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.556537+0000 front 2024-03-18T16:13:46.556598+0000 (oldest deadline 2024-03-18T16:14:11.856607+0000) 2024-03-18T16:14:13.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:13.668+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:13.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:13.668+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:13.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:13.668+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:13.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:13.668+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:13.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:13.668+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:14.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:14.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:14.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:14.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:14.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:14.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:14.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:13 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:13.891+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:14.508 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:14 smithi150 podman[91188]: 2024-03-18 16:14:14.222181451 +0000 UTC m=+5.544174463 container died dde576b5a28032bb7bd1037b55d27d3d3e1c68ed6395a45771bcc13038db8b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T16:14:14.765 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:14 smithi150 podman[91188]: 2024-03-18 16:14:14.589556301 +0000 UTC m=+5.911549304 container cleanup dde576b5a28032bb7bd1037b55d27d3d3e1c68ed6395a45771bcc13038db8b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T16:14:14.765 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:14 smithi150 bash[91188]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9 2024-03-18T16:14:14.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:14 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:14.704+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:14.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:14 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:14.704+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:14.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:14 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:14.704+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:14.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:14 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:14.704+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:14.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:14 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:14.704+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:15.092 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:14 smithi150 podman[91204]: 2024-03-18 16:14:14.891163636 +0000 UTC m=+0.668584835 container remove dde576b5a28032bb7bd1037b55d27d3d3e1c68ed6395a45771bcc13038db8b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:15.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:15.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:14 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:14.929+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:15.545 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.144775685 +0000 UTC m=+0.018377116 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:14:15.545 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.250352148 +0000 UTC m=+0.123953579 container create d41819cb1704757198ab2c32e093090080018f13981f345c7e81b0b0eefab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T16:14:15.545 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.461058139 +0000 UTC m=+0.334659574 container init d41819cb1704757198ab2c32e093090080018f13981f345c7e81b0b0eefab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:14:15.545 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.464048654 +0000 UTC m=+0.337650092 container start d41819cb1704757198ab2c32e093090080018f13981f345c7e81b0b0eefab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-18T16:14:15.546 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.544215886 +0000 UTC m=+0.417817335 container attach d41819cb1704757198ab2c32e093090080018f13981f345c7e81b0b0eefab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T16:14:15.843 INFO:journalctl@ceph.osd.9.smithi150.stdout:Mar 18 16:14:15 smithi150 podman[91294]: 2024-03-18 16:14:15.753552036 +0000 UTC m=+0.627153469 container died d41819cb1704757198ab2c32e093090080018f13981f345c7e81b0b0eefab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T16:14:15.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:15.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:15.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:15.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:15.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:15.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:15 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:15.658+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:16.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:15 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:15.886+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:16.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T16:14:16.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:16.357 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T16:14:16.357 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T16:14:16.357 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.10 2024-03-18T16:14:16.661 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 systemd[1]: Stopping Ceph osd.10 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:16.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:16.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:16.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:16.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:16.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:16.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:16 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:16.648+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.658+0000 7fd04e7b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.658+0000 7fd04e7b3700 -1 osd.10 106 *** Got signal Terminated *** 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.658+0000 7fd04e7b3700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:17.019 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:17.020 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:17.020 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:17.020 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:16 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:16.859+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:17.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:17.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:17.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:17.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:17.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:17.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:17 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:17.657+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:18.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:18.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:18.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:18.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:18.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:18.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:18.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:17 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:17.890+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:18.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:18.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:18.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:18.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:18.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:18.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:18.619+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:19.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:19.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:19.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:19.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:19.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:19.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:19.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:18 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:18.934+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:19.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:19.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:19.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:19.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:19.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:19.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:19 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:19.605+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:20.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:20.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:20.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:20.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:20.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:20.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:20.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:19 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:19.910+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:20.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:20.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:20.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:20.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:20.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:20.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:20.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:20 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:20.590+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:21.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:12:58.156499+0000 front 2024-03-18T16:12:58.156432+0000 (oldest deadline 2024-03-18T16:13:24.056505+0000) 2024-03-18T16:14:21.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:04.057013+0000 front 2024-03-18T16:13:04.056904+0000 (oldest deadline 2024-03-18T16:13:35.257640+0000) 2024-03-18T16:14:21.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:16.358681+0000 front 2024-03-18T16:13:16.358608+0000 (oldest deadline 2024-03-18T16:13:38.058711+0000) 2024-03-18T16:14:21.269 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6814 osd.3 since back 2024-03-18T16:13:24.360986+0000 front 2024-03-18T16:13:24.360926+0000 (oldest deadline 2024-03-18T16:13:47.260714+0000) 2024-03-18T16:14:21.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:31.961745+0000 front 2024-03-18T16:13:31.961784+0000 (oldest deadline 2024-03-18T16:13:56.661816+0000) 2024-03-18T16:14:21.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:38.362630+0000 front 2024-03-18T16:13:38.362694+0000 (oldest deadline 2024-03-18T16:14:02.462645+0000) 2024-03-18T16:14:21.270 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:20 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10[60967]: 2024-03-18T16:14:20.925+0000 7fd04ada7700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:46.563441+0000 front 2024-03-18T16:13:46.563410+0000 (oldest deadline 2024-03-18T16:14:11.263451+0000) 2024-03-18T16:14:21.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:21.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:21 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:21.594+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:22.303 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:22 smithi156 podman[73668]: 2024-03-18 16:14:22.043103444 +0000 UTC m=+5.495162068 container died cac2552c94fcc8bab7b32a1c0835303bdb3d87cfac8cf920ab9a3cc055fb256c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T16:14:22.578 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:22 smithi156 podman[73668]: 2024-03-18 16:14:22.376133153 +0000 UTC m=+5.828191784 container cleanup cac2552c94fcc8bab7b32a1c0835303bdb3d87cfac8cf920ab9a3cc055fb256c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T16:14:22.579 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:22 smithi156 bash[73668]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:22.917 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:22 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:22.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:22.981 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:22 smithi156 podman[73685]: 2024-03-18 16:14:22.577434834 +0000 UTC m=+0.528637757 container remove cac2552c94fcc8bab7b32a1c0835303bdb3d87cfac8cf920ab9a3cc055fb256c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T16:14:23.265 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:22 smithi156 podman[73772]: 2024-03-18 16:14:22.883108218 +0000 UTC m=+0.017842081 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:14:23.266 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:23 smithi156 podman[73772]: 2024-03-18 16:14:23.011881942 +0000 UTC m=+0.146615824 container create 9c16c265138c01f0be51b4f59a562021fefdddb1c0757c52737aba1ed81c1907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T16:14:23.266 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:23 smithi156 podman[73772]: 2024-03-18 16:14:23.205606208 +0000 UTC m=+0.340340072 container init 9c16c265138c01f0be51b4f59a562021fefdddb1c0757c52737aba1ed81c1907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T16:14:23.266 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:23 smithi156 podman[73772]: 2024-03-18 16:14:23.208644253 +0000 UTC m=+0.343378121 container start 9c16c265138c01f0be51b4f59a562021fefdddb1c0757c52737aba1ed81c1907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:14:23.266 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:23 smithi156 podman[73772]: 2024-03-18 16:14:23.263662194 +0000 UTC m=+0.398396067 container attach 9c16c265138c01f0be51b4f59a562021fefdddb1c0757c52737aba1ed81c1907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T16:14:23.519 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 18 16:14:23 smithi156 podman[73772]: 2024-03-18 16:14:23.498528246 +0000 UTC m=+0.633262110 container died 9c16c265138c01f0be51b4f59a562021fefdddb1c0757c52737aba1ed81c1907 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T16:14:23.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:23.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:23.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:23.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:23.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:23.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:23.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:23 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:23.587+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:24.074 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T16:14:24.116 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:24.117 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T16:14:24.117 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T16:14:24.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.11 2024-03-18T16:14:24.399 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 systemd[1]: Stopping Ceph osd.11 for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:24.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.397+0000 7fc367c05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:24.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.397+0000 7fc367c05700 -1 osd.11 106 *** Got signal Terminated *** 2024-03-18T16:14:24.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.397+0000 7fc367c05700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:14:24.713 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:24.714 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:24 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:24.564+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:25.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:25.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:25.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:25.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:25.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:25.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:25.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:25 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:25.557+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:26.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:26.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:26.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:26.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:26.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:26.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:26.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:26 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:26.575+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:27.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:27.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:27.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:27 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:27.555+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T16:14:02.549289+0000 front 2024-03-18T16:14:02.549331+0000 (oldest deadline 2024-03-18T16:14:26.649343+0000) 2024-03-18T16:14:28.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6806 osd.0 since back 2024-03-18T16:13:00.738707+0000 front 2024-03-18T16:13:00.738842+0000 (oldest deadline 2024-03-18T16:13:24.838821+0000) 2024-03-18T16:14:28.963 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-18T16:13:08.940203+0000 front 2024-03-18T16:13:08.939951+0000 (oldest deadline 2024-03-18T16:13:33.640085+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2024-03-18T16:13:15.341191+0000 front 2024-03-18T16:13:15.341216+0000 (oldest deadline 2024-03-18T16:13:39.942124+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-18T16:13:33.344750+0000 front 2024-03-18T16:13:33.344907+0000 (oldest deadline 2024-03-18T16:13:56.244855+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2024-03-18T16:13:40.846248+0000 front 2024-03-18T16:13:40.846287+0000 (oldest deadline 2024-03-18T16:14:04.946415+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.150:6822 osd.6 since back 2024-03-18T16:13:44.947019+0000 front 2024-03-18T16:13:44.947055+0000 (oldest deadline 2024-03-18T16:14:14.947421+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-18T16:13:54.948155+0000 front 2024-03-18T16:13:54.948185+0000 (oldest deadline 2024-03-18T16:14:19.648177+0000) 2024-03-18T16:14:28.964 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:28 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11[62601]: 2024-03-18T16:14:28.585+0000 7fc3641f9700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.191:6820 osd.8 since back 2024-03-18T16:14:02.549289+0000 front 2024-03-18T16:14:02.549331+0000 (oldest deadline 2024-03-18T16:14:26.649343+0000) 2024-03-18T16:14:30.212 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:29 smithi191 podman[75378]: 2024-03-18 16:14:29.921944688 +0000 UTC m=+5.613292735 container died aade0d9cc405690d9d9b095bbde8e99ee3214408d9c2d5149a4415ba6d6ce6dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T16:14:30.463 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:30 smithi191 podman[75378]: 2024-03-18 16:14:30.313992065 +0000 UTC m=+6.005340115 container cleanup aade0d9cc405690d9d9b095bbde8e99ee3214408d9c2d5149a4415ba6d6ce6dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:14:30.463 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:30 smithi191 bash[75378]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11 2024-03-18T16:14:30.859 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:30 smithi191 podman[75395]: 2024-03-18 16:14:30.565441992 +0000 UTC m=+0.641621214 container remove aade0d9cc405690d9d9b095bbde8e99ee3214408d9c2d5149a4415ba6d6ce6dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T16:14:31.136 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:30 smithi191 podman[75488]: 2024-03-18 16:14:30.857777443 +0000 UTC m=+0.104038924 container create b1ef028db7e1a0495432c5255eb176a405098c7a1954c755c92e08064d9ece27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T16:14:31.136 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:30 smithi191 podman[75488]: 2024-03-18 16:14:30.771373438 +0000 UTC m=+0.017634900 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:14:31.136 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:31 smithi191 podman[75488]: 2024-03-18 16:14:31.060230658 +0000 UTC m=+0.306492129 container init b1ef028db7e1a0495432c5255eb176a405098c7a1954c755c92e08064d9ece27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:14:31.136 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:31 smithi191 podman[75488]: 2024-03-18 16:14:31.063281578 +0000 UTC m=+0.309543049 container start b1ef028db7e1a0495432c5255eb176a405098c7a1954c755c92e08064d9ece27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T16:14:31.136 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:31 smithi191 podman[75488]: 2024-03-18 16:14:31.134870539 +0000 UTC m=+0.381132012 container attach b1ef028db7e1a0495432c5255eb176a405098c7a1954c755c92e08064d9ece27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:14:31.463 INFO:journalctl@ceph.osd.11.smithi191.stdout:Mar 18 16:14:31 smithi191 podman[75488]: 2024-03-18 16:14:31.348498924 +0000 UTC m=+0.594760393 container died b1ef028db7e1a0495432c5255eb176a405098c7a1954c755c92e08064d9ece27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:14:31.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T16:14:31.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:31.948 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T16:14:31.949 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T16:14:31.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.a 2024-03-18T16:14:32.342 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:31 smithi150 systemd[1]: Stopping Ceph mds.a for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:32.842 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:32 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a[71461]: 2024-03-18T16:14:32.375+0000 7f25f2f9a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:32.843 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:32 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a[71461]: 2024-03-18T16:14:32.375+0000 7f25f2f9a700 -1 mds.a *** got signal Terminated *** 2024-03-18T16:14:36.592 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:36 smithi150 podman[91411]: 2024-03-18 16:14:36.259574646 +0000 UTC m=+4.121557932 container died 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T16:14:36.868 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:36 smithi150 podman[91411]: 2024-03-18 16:14:36.666363855 +0000 UTC m=+4.528347150 container cleanup 8e509fcc0158acc40582e2e8b32850481c8985c61f8f8dc6b6edcf6779f0ca31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:14:36.869 INFO:journalctl@ceph.mds.a.smithi150.stdout:Mar 18 16:14:36 smithi150 bash[91411]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-a 2024-03-18T16:14:37.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T16:14:37.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:37.050 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T16:14:37.050 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T16:14:37.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.d 2024-03-18T16:14:37.592 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:37 smithi150 systemd[1]: Stopping Ceph mds.d for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:37.593 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d[70478]: 2024-03-18T16:14:37.378+0000 7f69d5aae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:37.593 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:37 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d[70478]: 2024-03-18T16:14:37.378+0000 7f69d5aae700 -1 mds.d *** got signal Terminated *** 2024-03-18T16:14:40.092 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:39 smithi150 podman[91544]: 2024-03-18 16:14:39.817847233 +0000 UTC m=+2.550147675 container died 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T16:14:40.547 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:40 smithi150 podman[91544]: 2024-03-18 16:14:40.145122482 +0000 UTC m=+2.877422924 container cleanup 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T16:14:40.548 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:40 smithi150 bash[91544]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d 2024-03-18T16:14:40.548 INFO:journalctl@ceph.mds.d.smithi150.stdout:Mar 18 16:14:40 smithi150 podman[91559]: 2024-03-18 16:14:40.329911171 +0000 UTC m=+0.514212501 container remove 033a22e8745d2772e94a89755da70c32e6001be07fab7af7f89b49906782cd79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True) 2024-03-18T16:14:40.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T16:14:40.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:40.606 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T16:14:40.607 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T16:14:40.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.g 2024-03-18T16:14:40.842 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:40 smithi150 systemd[1]: Stopping Ceph mds.g for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:41.342 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g[71084]: 2024-03-18T16:14:40.932+0000 7f2c83d5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:41.343 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:40 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g[71084]: 2024-03-18T16:14:40.932+0000 7f2c83d5b700 -1 mds.g *** got signal Terminated *** 2024-03-18T16:14:45.217 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:44 smithi150 podman[91665]: 2024-03-18 16:14:44.945842217 +0000 UTC m=+4.120306066 container died ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T16:14:45.584 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:45 smithi150 podman[91665]: 2024-03-18 16:14:45.323959193 +0000 UTC m=+4.498423037 container cleanup ddd2d6b72776ea9393a663259e8b5d85f6a51b1b2eb622c07de57d41812fee27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-18T16:14:45.585 INFO:journalctl@ceph.mds.g.smithi150.stdout:Mar 18 16:14:45 smithi150 bash[91665]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-g 2024-03-18T16:14:45.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T16:14:45.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:45.775 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T16:14:45.775 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T16:14:45.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.j 2024-03-18T16:14:46.092 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:45 smithi150 systemd[1]: Stopping Ceph mds.j for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:46.093 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j[70131]: 2024-03-18T16:14:46.085+0000 7fa5d0a19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:46.093 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:46 smithi150 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j[70131]: 2024-03-18T16:14:46.085+0000 7fa5d0a19700 -1 mds.j *** got signal Terminated *** 2024-03-18T16:14:49.304 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:48 smithi150 podman[91787]: 2024-03-18 16:14:48.994331025 +0000 UTC m=+3.001860969 container died 828b49adc05d07e7fd5fa2a479b42f629a23d84af249b4caa4a47b828e86120c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T16:14:49.304 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:49 smithi150 podman[91787]: 2024-03-18 16:14:49.302743881 +0000 UTC m=+3.310273823 container cleanup 828b49adc05d07e7fd5fa2a479b42f629a23d84af249b4caa4a47b828e86120c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T16:14:49.592 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:49 smithi150 bash[91787]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j 2024-03-18T16:14:49.593 INFO:journalctl@ceph.mds.j.smithi150.stdout:Mar 18 16:14:49 smithi150 podman[91801]: 2024-03-18 16:14:49.520722833 +0000 UTC m=+0.524344682 container remove 828b49adc05d07e7fd5fa2a479b42f629a23d84af249b4caa4a47b828e86120c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T16:14:49.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T16:14:49.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:49.694 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T16:14:49.694 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T16:14:49.694 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.b 2024-03-18T16:14:50.019 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:49 smithi156 systemd[1]: Stopping Ceph mds.b for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:50.518 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: 2024-03-18T16:14:50.101+0000 7f35c236f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:50.519 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:50 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b[66367]: 2024-03-18T16:14:50.101+0000 7f35c236f700 -1 mds.b *** got signal Terminated *** 2024-03-18T16:14:53.222 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:52 smithi156 podman[73888]: 2024-03-18 16:14:52.889730605 +0000 UTC m=+3.003686616 container died 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T16:14:53.495 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:53 smithi156 podman[73888]: 2024-03-18 16:14:53.31733005 +0000 UTC m=+3.431286041 container cleanup 03004a5dc05892d47496633c7e0fd7bd217966f0ada115302f5320a760f8aae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:14:53.496 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 18 16:14:53 smithi156 bash[73888]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-b 2024-03-18T16:14:53.698 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T16:14:53.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:53.737 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T16:14:53.737 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T16:14:53.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.e 2024-03-18T16:14:54.056 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:53 smithi156 systemd[1]: Stopping Ceph mds.e for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:54.519 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: 2024-03-18T16:14:54.054+0000 7f38c345b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:54.519 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:54 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e[65736]: 2024-03-18T16:14:54.054+0000 7f38c345b700 -1 mds.e *** got signal Terminated *** 2024-03-18T16:14:57.248 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:56 smithi156 podman[74007]: 2024-03-18 16:14:56.894699262 +0000 UTC m=+2.931361916 container died 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T16:14:57.518 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:57 smithi156 podman[74007]: 2024-03-18 16:14:57.346251144 +0000 UTC m=+3.382913789 container cleanup 412fb943d569f98992a1d01e02d55d89a288c11ff42f18846e596f104ffcec84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True) 2024-03-18T16:14:57.519 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 18 16:14:57 smithi156 bash[74007]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-e 2024-03-18T16:14:57.758 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T16:14:57.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:14:57.796 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T16:14:57.796 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T16:14:57.796 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.h 2024-03-18T16:14:58.127 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:14:57 smithi156 systemd[1]: Stopping Ceph mds.h for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:14:58.519 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:14:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: 2024-03-18T16:14:58.124+0000 7f3295073700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:14:58.519 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:14:58 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h[66045]: 2024-03-18T16:14:58.124+0000 7f3295073700 -1 mds.h *** got signal Terminated *** 2024-03-18T16:15:01.152 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:15:00 smithi156 podman[74140]: 2024-03-18 16:15:00.892070682 +0000 UTC m=+2.878317538 container died 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-18T16:15:01.519 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:15:01 smithi156 podman[74140]: 2024-03-18 16:15:01.250334869 +0000 UTC m=+3.236581741 container cleanup 27d513d08cb8bc72665e9e45310dd30d4724d8ea1ac9be545aff7b79dba83726 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T16:15:01.519 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 18 16:15:01 smithi156 bash[74140]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-h 2024-03-18T16:15:01.637 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T16:15:01.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:01.671 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T16:15:01.672 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T16:15:01.672 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.k 2024-03-18T16:15:01.997 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:01 smithi156 systemd[1]: Stopping Ceph mds.k for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:15:02.268 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k[65409]: 2024-03-18T16:15:01.995+0000 7f695f275700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:15:02.269 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:01 smithi156 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k[65409]: 2024-03-18T16:15:01.995+0000 7f695f275700 -1 mds.k *** got signal Terminated *** 2024-03-18T16:15:06.256 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 podman[74264]: 2024-03-18 16:15:06.000720406 +0000 UTC m=+4.111843387 container died 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T16:15:06.519 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 podman[74264]: 2024-03-18 16:15:06.34552015 +0000 UTC m=+4.456643135 container cleanup 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-18T16:15:06.519 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 bash[74264]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k 2024-03-18T16:15:06.762 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T16:15:06.785 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 podman[74278]: 2024-03-18 16:15:06.522161692 +0000 UTC m=+0.515159278 container remove 85c0e1a3757035b92dbaff0683a11110b9d9ef3d775014d4a3b8638823ef5a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T16:15:06.786 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-18T16:15:06.786 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 systemd[1]: Stopped Ceph mds.k for 0cd032e0-e53f-11ee-95c9-87774f69a715. 2024-03-18T16:15:06.786 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 18 16:15:06 smithi156 systemd[1]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.k.service: Consumed 1.023s CPU time. 2024-03-18T16:15:06.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:06.797 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T16:15:06.797 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T16:15:06.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.c 2024-03-18T16:15:07.105 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:06 smithi191 systemd[1]: Stopping Ceph mds.c for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:15:07.367 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: 2024-03-18T16:15:07.102+0000 7fe516e5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:15:07.368 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:07 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c[67623]: 2024-03-18T16:15:07.103+0000 7fe516e5b700 -1 mds.c *** got signal Terminated *** 2024-03-18T16:15:09.213 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:08 smithi191 podman[75592]: 2024-03-18 16:15:08.88816585 +0000 UTC m=+1.898060476 container died 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T16:15:09.572 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:09 smithi191 podman[75592]: 2024-03-18 16:15:09.319551974 +0000 UTC m=+2.329446590 container cleanup 4608e976db80cee96c7883655546238432873c0422862f0e61ab8ca3f545c21e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:15:09.572 INFO:journalctl@ceph.mds.c.smithi191.stdout:Mar 18 16:15:09 smithi191 bash[75592]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-c 2024-03-18T16:15:09.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T16:15:09.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:09.739 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T16:15:09.740 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T16:15:09.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f 2024-03-18T16:15:09.963 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:09 smithi191 systemd[1]: Stopping Ceph mds.f for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:15:10.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[72144]: 2024-03-18T16:15:10.072+0000 7f7a69283700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:15:10.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:10 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f[72144]: 2024-03-18T16:15:10.072+0000 7f7a69283700 -1 mds.f *** got signal Terminated *** 2024-03-18T16:15:13.059 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:12 smithi191 podman[75727]: 2024-03-18 16:15:12.807540601 +0000 UTC m=+2.854383493 container died 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T16:15:13.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:13 smithi191 podman[75727]: 2024-03-18 16:15:13.13121681 +0000 UTC m=+3.178059706 container cleanup 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T16:15:13.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:13 smithi191 bash[75727]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f 2024-03-18T16:15:13.463 INFO:journalctl@ceph.mds.f.smithi191.stdout:Mar 18 16:15:13 smithi191 podman[75741]: 2024-03-18 16:15:13.307588621 +0000 UTC m=+0.495355869 container remove 593deb3ee40f17ca22f2aea98934f39ad9ef7a4feb13525330e2f0f2cb4d56f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T16:15:13.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T16:15:13.554 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:13.555 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T16:15:13.555 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T16:15:13.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.i 2024-03-18T16:15:13.870 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:13 smithi191 systemd[1]: Stopping Ceph mds.i for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:15:14.213 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: 2024-03-18T16:15:13.866+0000 7f0e46830700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:15:14.213 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:13 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i[67935]: 2024-03-18T16:15:13.866+0000 7f0e46830700 -1 mds.i *** got signal Terminated *** 2024-03-18T16:15:17.162 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:16 smithi191 podman[75848]: 2024-03-18 16:15:16.881776454 +0000 UTC m=+3.119711191 container died ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T16:15:17.463 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:17 smithi191 podman[75848]: 2024-03-18 16:15:17.276893003 +0000 UTC m=+3.514827721 container cleanup ad5640ce5aab8ed1b957b35866eeba7ad33f17147bc6613d24afed8dbafdcbbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T16:15:17.463 INFO:journalctl@ceph.mds.i.smithi191.stdout:Mar 18 16:15:17 smithi191 bash[75848]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-i 2024-03-18T16:15:17.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T16:15:17.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:17.693 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T16:15:17.694 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T16:15:17.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.l 2024-03-18T16:15:17.786 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:17 smithi191 systemd[1]: Stopping Ceph mds.l for 0cd032e0-e53f-11ee-95c9-87774f69a715... 2024-03-18T16:15:18.213 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: 2024-03-18T16:15:18.012+0000 7f06f5b62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:15:18.213 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:18 smithi191 ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l[66999]: 2024-03-18T16:15:18.012+0000 7f06f5b62700 -1 mds.l *** got signal Terminated *** 2024-03-18T16:15:21.213 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:20 smithi191 podman[75969]: 2024-03-18 16:15:20.955705456 +0000 UTC m=+3.049071514 container died b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T16:15:21.574 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:21 smithi191 podman[75969]: 2024-03-18 16:15:21.330424726 +0000 UTC m=+3.423790784 container cleanup b1298fd4c56e17c15fd81398a74b315eecf457ae6e4941e92a958995285e67da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-18T16:15:21.575 INFO:journalctl@ceph.mds.l.smithi191.stdout:Mar 18 16:15:21 smithi191 bash[75969]: ceph-0cd032e0-e53f-11ee-95c9-87774f69a715-mds-l 2024-03-18T16:15:21.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0cd032e0-e53f-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T16:15:21.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:15:21.725 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T16:15:21.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:15:21.917 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:15:24.440 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:15:24.632 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:15:27.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:15:27.390 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:15:29.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:15:29.982 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:15:30.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:15:30.046 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T16:15:30.048 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi150/crash 2024-03-18T16:15:30.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:15:30.115 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:15:30.116 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:15:30.117 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi156/crash 2024-03-18T16:15:30.118 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:15:30.163 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:15:30.163 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:15:30.165 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi191/crash 2024-03-18T16:15:30.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:15:30.208 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:15:30.208 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:15:30.209 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T16:15:30.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-18T16:15:30.253 INFO:teuthology.orchestra.run.smithi150.stdout:2024-03-18T16:04:57.463510+0000 mds.i (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi150:x (15312), after 300.543 seconds 2024-03-18T16:15:30.254 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T16:15:30.254 INFO:tasks.cephadm:Compressing logs... 2024-03-18T16:15:30.254 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:15:30.296 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:15:30.298 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:15:30.321 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T16:15:30.322 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2024-03-18T16:15:30.323 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T16:15:30.323 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T16:15:30.323 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:15:30.325 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T16:15:30.325 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T16:15:30.325 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T16:15:30.325 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T16:15:30.325 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T16:15:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.4% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:15:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:15:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T16:15:30.329 INFO:teuthology.orchestra.run.smithi150.stderr: 94.7% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T16:15:30.332 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory 2024-03-18T16:15:30.332 INFO:teuthology.orchestra.run.smithi156.stderr: /var/log/ceph/cephadm.log 2024-03-18T16:15:30.332 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T16:15:30.332 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.b.log: --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T16:15:30.333 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.5% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:15:30.334 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T16:15:30.336 INFO:teuthology.orchestra.run.smithi150.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:15:30.336 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:15:30.336 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T16:15:30.336 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log: 84.2% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:15:30.337 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T16:15:30.337 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log: 87.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:15:30.337 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T16:15:30.337 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T16:15:30.339 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T16:15:30.339 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T16:15:30.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T16:15:30.344 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T16:15:30.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T16:15:30.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T16:15:30.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.c.log: 92.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T16:15:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T16:15:30.347 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.y.log: 91.9% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T16:15:30.347 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:15:30.347 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:15:30.348 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T16:15:30.348 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:15:30.351 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log: 87.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:15:30.351 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T16:15:30.351 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.7% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T16:15:30.365 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:15:30.378 INFO:teuthology.orchestra.run.smithi191.stderr: 83.8% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:15:30.396 INFO:teuthology.orchestra.run.smithi156.stderr: 92.3% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T16:15:30.524 INFO:teuthology.orchestra.run.smithi191.stderr: 90.5% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:15:30.668 INFO:teuthology.orchestra.run.smithi150.stderr: 89.8% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T16:15:30.830 INFO:teuthology.orchestra.run.smithi156.stderr: 91.9% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T16:15:30.862 INFO:teuthology.orchestra.run.smithi191.stderr: 90.9% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:15:30.947 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:15:30.950 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:15:31.281 INFO:teuthology.orchestra.run.smithi150.stderr: 90.3% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T16:15:32.024 INFO:teuthology.orchestra.run.smithi156.stderr: 95.2% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T16:15:35.456 INFO:teuthology.orchestra.run.smithi191.stderr: 94.1% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T16:15:36.122 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T16:15:41.234 INFO:teuthology.orchestra.run.smithi150.stderr: 91.9% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T16:15:41.435 INFO:teuthology.orchestra.run.smithi150.stderr: 91.4% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T16:15:41.831 INFO:teuthology.orchestra.run.smithi156.stderr: 91.8% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T16:15:42.856 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T16:15:43.811 INFO:teuthology.orchestra.run.smithi156.stderr: 91.3% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T16:15:44.888 INFO:teuthology.orchestra.run.smithi156.stderr: 91.2% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T16:15:45.264 INFO:teuthology.orchestra.run.smithi150.stderr: 91.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T16:15:45.267 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-18T16:15:45.267 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m14.957s 2024-03-18T16:15:45.267 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m49.240s 2024-03-18T16:15:45.267 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m1.527s 2024-03-18T16:15:49.092 INFO:teuthology.orchestra.run.smithi156.stderr: 91.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T16:15:56.804 INFO:teuthology.orchestra.run.smithi156.stderr: 94.4% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T16:15:56.805 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-18T16:15:56.806 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m26.493s 2024-03-18T16:15:56.806 INFO:teuthology.orchestra.run.smithi156.stderr:user 1m24.221s 2024-03-18T16:15:56.806 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m2.667s 2024-03-18T16:16:01.705 INFO:teuthology.orchestra.run.smithi191.stderr: 95.8% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T16:16:08.771 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T16:16:11.082 INFO:teuthology.orchestra.run.smithi191.stderr: 91.3% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T16:16:12.165 INFO:teuthology.orchestra.run.smithi191.stderr: 95.6% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T16:16:19.356 INFO:teuthology.orchestra.run.smithi191.stderr: 91.0% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T16:16:19.721 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T16:19:38.623 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/0cd032e0-e53f-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T16:19:38.625 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-18T16:19:38.625 INFO:teuthology.orchestra.run.smithi191.stderr:real 4m8.313s 2024-03-18T16:19:38.626 INFO:teuthology.orchestra.run.smithi191.stderr:user 4m31.734s 2024-03-18T16:19:38.626 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m9.907s 2024-03-18T16:19:38.626 INFO:tasks.cephadm:Archiving logs... 2024-03-18T16:19:38.627 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi150/log 2024-03-18T16:19:38.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:19:46.903 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi156/log 2024-03-18T16:19:46.905 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:19:58.827 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi191/log 2024-03-18T16:19:58.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:20:58.263 INFO:tasks.cephadm:Removing cluster... 2024-03-18T16:20:58.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force 2024-03-18T16:20:58.461 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:20:58.795 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force 2024-03-18T16:20:58.988 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:20:59.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0cd032e0-e53f-11ee-95c9-87774f69a715 --force 2024-03-18T16:20:59.544 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: 0cd032e0-e53f-11ee-95c9-87774f69a715 2024-03-18T16:21:00.079 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T16:21:00.080 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:21:00.096 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:21:00.113 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:21:00.163 INFO:tasks.cephadm:Teardown complete 2024-03-18T16:21:00.164 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T16:21:00.175 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T16:21:00.175 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:21:00.177 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:21:00.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:21:00.323 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T16:21:00.333 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T16:21:00.334 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:21:00.336 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:21:00.339 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:21:00.348 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:21:00.430 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:21:00.431 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +2506us[+2506us] +/- 90ms 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1484us[-1478us] +/- 69ms 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -332us[ -332us] +/- 63ms 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:21:00.433 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +2498us[+2467us] +/- 90ms 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1500us[-1532us] +/- 69ms 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -339us[ -339us] +/- 63ms 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:21:00.434 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:21:00.442 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-18T16:21:00.442 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +2517us[+2524us] +/- 90ms 2024-03-18T16:21:00.442 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -1496us[-1489us] +/- 69ms 2024-03-18T16:21:00.442 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -339us[ -339us] +/- 63ms 2024-03-18T16:21:00.443 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:21:00.443 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T16:21:00.454 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T16:21:00.454 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T16:21:00.499 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:21:00.595 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:21:00.704 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:21:00.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:21:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:00.830 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:21:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:00.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:21:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:00.966 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T16:21:00.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T16:21:00.989 INFO:teuthology.task.internal:Duration was 2334.933216 seconds 2024-03-18T16:21:00.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T16:21:01.009 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T16:21:01.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:21:01.011 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:21:01.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:21:01.052 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:21:01.057 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:21:01.081 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:21:01.553 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T16:21:01.553 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-03-18T16:21:01.581 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:21:01.600 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-03-18T16:21:01.600 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:21:01.628 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-18T16:21:01.630 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:21:01.659 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T16:21:01.659 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:21:01.662 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:21:01.670 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:21:01.728 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T16:21:01.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:21:01.731 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:21:01.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:21:02.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T16:21:02.114 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T16:21:02.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:21:02.141 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:21:02.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:21:02.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T16:21:02.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.243 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-03-18T16:21:02.245 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-03-18T16:21:02.272 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-18T16:21:02.293 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:02.390 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:02.412 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:21:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:21:02.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T16:21:02.439 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T16:21:02.439 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi150 2024-03-18T16:21:02.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:21:02.482 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi156 2024-03-18T16:21:02.545 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:21:02.621 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905/remote/smithi191 2024-03-18T16:21:02.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:21:02.659 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T16:21:02.659 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:21:02.662 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:21:02.664 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:21:02.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T16:21:02.729 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T16:21:02.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T16:21:02.779 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T16:21:02.779 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:21:02.781 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:21:02.783 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:21:02.801 INFO:teuthology.orchestra.run.smithi150.stdout: 265570 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:21 /home/ubuntu/cephtest 2024-03-18T16:21:02.801 INFO:teuthology.orchestra.run.smithi156.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:21 /home/ubuntu/cephtest 2024-03-18T16:21:02.813 INFO:teuthology.orchestra.run.smithi191.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:21 /home/ubuntu/cephtest 2024-03-18T16:21:02.814 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T16:21:02.852 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T16:21:02.946 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T16:21:02.973 DEBUG:teuthology.nuke:shortname: smithi150 2024-03-18T16:21:02.974 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:21:02.998 DEBUG:teuthology.nuke:shortname: smithi156 2024-03-18T16:21:02.999 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:21:03.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.191917', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWnDLCEvrTGwexuUVs+s2aEBzskPO31yExEk/1HVtP2AMtyfWTvwDCuQ20GIOpbUaCcZTpsorSh3uQhG7Hk6Pw='} 2024-03-18T16:21:03.023 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-18T16:21:03.024 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:21:03.044 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.190322', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqo8HF0mk5hVSYWkfZ0kSgaCgVo0YQEc2mLFsYAhEzuKq1GtgSJBUyWPbCBfhCKPF5asZwMUoY3PuJfNSJUvlg='} 2024-03-18T16:21:03.048 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608905', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:33:21.188733', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvpS9p7oSdf1WzK50BNQGRK/xrBY1V9Pjr5s+PsteXPEGx21njDYhacZi6D6ksPvMGCOG2PIn9jTZnmzTvg/Pc='} 2024-03-18T16:21:03.075 INFO:teuthology.orchestra.console.smithi150:Power off 2024-03-18T16:21:03.076 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:21:03.101 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-03-18T16:21:03.101 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:03.121 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-18T16:21:03.121 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:21:03.137 INFO:teuthology.orchestra.console.smithi156:Power off 2024-03-18T16:21:03.137 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:21:03.146 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-18T16:21:03.146 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-18T16:21:03.146 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:03.161 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-03-18T16:21:03.161 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:03.172 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T16:21:03.186 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-18T16:21:07.147 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:07.176 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:07.189 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:07.258 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-18T16:21:07.289 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T16:21:07.338 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-18T16:21:11.269 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:11.291 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:11.339 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:11.783 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-18T16:21:11.784 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-18T16:21:11.883 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-03-18T16:21:11.884 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-03-18T16:21:11.984 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-03-18T16:21:15.789 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:21:15.901 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-18T16:21:16.002 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-18T16:21:16.109 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2334.9332156181335 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi150 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=32e48465455c44c29ff08564d4078e54 status: fail success: false 2024-03-18T16:21:16.110 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T16:21:16.213 INFO:teuthology.run:FAIL